[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:01 dist-test-slave-fkbn krb5kdc[2214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:01 dist-test-slave-fkbn krb5kdc[2214](info): set up 2 sockets
Sep 26 06:25:01 dist-test-slave-fkbn krb5kdc[2214](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 06:25:02.774264 2207 mini_kdc.cc:121] Time spent starting KDC: real 3.395s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 06:25:02.994565 2207 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.220s user 0.002s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 06:25:03.105633 2207 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.111s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:03.188484 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36067
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:03.292313 2227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:03.292631 2227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:03.292677 2227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:03.294276 2227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:03.294318 2227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:03.294332 2227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:03.294345 2227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:03.294360 2227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:03.296094 2227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:03.296330 2227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:03.296566 2227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:03.301930 2232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:03.301915 2233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:03.301908 2235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:03.302211 2227 server_base.cc:1034] running on GCE node
I20260926 06:25:03.302448 2227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:03.302764 2227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:03.303931 2227 hybrid_clock.cc:648] HybridClock initialized: now 1790403903303928 us; error 32 us; skew 500 ppm
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:03.311453 2227 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:03.311941 2227 webserver.cc:460] Webserver started at http://127.2.39.254:42307/ using document root <none> and password file <none>
I20260926 06:25:03.312188 2227 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:03.312266 2227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:03.312407 2227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:03.313493 2227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "58dad1ebfc2b4847b515dba41ee7aa02"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "d599f5bd8bd1f4a3286404a3e7607a94"
server_key_iv: "e89c9d92fb099e35c63c1089cf7fb524"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.313902 2227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "58dad1ebfc2b4847b515dba41ee7aa02"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "d599f5bd8bd1f4a3286404a3e7607a94"
server_key_iv: "e89c9d92fb099e35c63c1089cf7fb524"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.315300 2227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:03.316045 2242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:03.316310 2227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:03.316416 2227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "58dad1ebfc2b4847b515dba41ee7aa02"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "d599f5bd8bd1f4a3286404a3e7607a94"
server_key_iv: "e89c9d92fb099e35c63c1089cf7fb524"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.316495 2227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:03.325889 2227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:03.328173 2227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:03.328333 2227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:03.332397 2227 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:36067
I20260926 06:25:03.332438 2294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:36067 every 8 connection(s)
I20260926 06:25:03.332836 2227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:03.333662 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 2227
I20260926 06:25:03.333768 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:03.333959 2295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:03.334535 2207 external_mini_cluster.cc:1428] Setting key ffb3df97a1fbde89024e2e89cd4a50be
I20260926 06:25:03.336913 2295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02: Bootstrap starting.
I20260926 06:25:03.337527 2295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:03.337962 2295 log.cc:824] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:03.338697 2295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02: No bootstrap required, opened a new log
I20260926 06:25:03.340790 2295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dad1ebfc2b4847b515dba41ee7aa02" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36067 } }
I20260926 06:25:03.340965 2295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:03.340998 2295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dad1ebfc2b4847b515dba41ee7aa02, State: Initialized, Role: FOLLOWER
I20260926 06:25:03.341151 2295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dad1ebfc2b4847b515dba41ee7aa02" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36067 } }
I20260926 06:25:03.341228 2295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:03.341255 2295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:03.341291 2295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:03.342002 2295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dad1ebfc2b4847b515dba41ee7aa02" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36067 } }
I20260926 06:25:03.342147 2295 leader_election.cc:302] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58dad1ebfc2b4847b515dba41ee7aa02; no voters:
I20260926 06:25:03.342350 2295 leader_election.cc:288] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:03.342382 2300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:03.342491 2300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [term 1 LEADER]: Becoming Leader. State: Replica: 58dad1ebfc2b4847b515dba41ee7aa02, State: Running, Role: LEADER
I20260926 06:25:03.342609 2295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:03.342582 2300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dad1ebfc2b4847b515dba41ee7aa02" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36067 } }
I20260926 06:25:03.342960 2300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58dad1ebfc2b4847b515dba41ee7aa02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dad1ebfc2b4847b515dba41ee7aa02" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36067 } } }
I20260926 06:25:03.343051 2300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:03.343186 2300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58dad1ebfc2b4847b515dba41ee7aa02. Latest consensus state: current_term: 1 leader_uuid: "58dad1ebfc2b4847b515dba41ee7aa02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58dad1ebfc2b4847b515dba41ee7aa02" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36067 } } }
I20260926 06:25:03.343266 2300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:03.343410 2306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:03.344043 2306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:03.345630 2306 catalog_manager.cc:1347] Generated new cluster ID: 39dfdd5ae7534fa28f90c280327b1de8
I20260926 06:25:03.345685 2306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:03.352005 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:03.335455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.2.39.254:36067)
0926 06:25:03.335635 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:03.335644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:03.335815 (+ 171us) server_negotiation.cc:407] Connection header received
0926 06:25:03.338292 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:03.338298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:03.338339 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:03.338530 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:03.340353 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.341919 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.343527 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.344155 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.344232 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:03.347984 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:03.347988 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:03.347988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:03.347999 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:03.349797 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:03.350367 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:03.350370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:03.350371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:03.350463 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:03.350781 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:03.350782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:03.350782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:03.351126 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:03.351360 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:03.351664 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 06:25:03.351806 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
I20260926 06:25:03.374548 2306 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:03.375171 2306 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:03.382932 2306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58dad1ebfc2b4847b515dba41ee7aa02: Generated new TSK 0
I20260926 06:25:03.383124 2306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:03.416531 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36067
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:03.526432 2321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:03.526691 2321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:03.526717 2321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:03.528023 2321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:03.528065 2321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:03.528097 2321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:03.529901 2321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:03.530133 2321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:03.530367 2321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:03.535765 2327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:03.535996 2321 server_base.cc:1034] running on GCE node
W20260926 06:25:03.535697 2329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:03.536177 2326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:03.536399 2321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:03.536684 2321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:03.537822 2321 hybrid_clock.cc:648] HybridClock initialized: now 1790403903537813 us; error 66 us; skew 500 ppm
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:03.540588 2321 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:03.541004 2321 webserver.cc:460] Webserver started at http://127.2.39.193:38905/ using document root <none> and password file <none>
I20260926 06:25:03.541273 2321 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:03.541340 2321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:03.541496 2321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:03.542744 2321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "4026a6600d6c4f17b9c9abdc2c5a4175"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "8cc4ca3390f6cf1459af374ea53d40cc"
server_key_iv: "23e6f7de49a184dd64c1838717f7c07e"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.543182 2321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "4026a6600d6c4f17b9c9abdc2c5a4175"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "8cc4ca3390f6cf1459af374ea53d40cc"
server_key_iv: "23e6f7de49a184dd64c1838717f7c07e"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.544766 2321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:03.545804 2336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:03.545984 2321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:03.546082 2321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "4026a6600d6c4f17b9c9abdc2c5a4175"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "8cc4ca3390f6cf1459af374ea53d40cc"
server_key_iv: "23e6f7de49a184dd64c1838717f7c07e"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.546176 2321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:03.574771 2321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:03.577399 2321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:03.577529 2321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:03.577847 2321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:03.578204 2321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:03.578262 2321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:03.578294 2321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:03.578325 2321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:03.585843 2321 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:43547
I20260926 06:25:03.585955 2449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:43547 every 8 connection(s)
I20260926 06:25:03.586287 2321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:03.592371 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 2321
I20260926 06:25:03.592484 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:03.592649 2207 external_mini_cluster.cc:1428] Setting key a6eee019badce53e73851d648f176ae6
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:03.598559 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:03.587049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:60891 (local address 127.2.39.254:36067)
0926 06:25:03.587210 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:03.587211 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:03.587431 (+ 220us) server_negotiation.cc:407] Connection header received
0926 06:25:03.588104 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:03.588105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:03.588114 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:03.588163 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:03.589307 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.589918 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.591222 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.591434 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.591530 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:03.594369 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:03.594371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:03.594371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:03.594379 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:03.596643 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:03.597343 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:03.597344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:03.597345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:03.597437 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:03.597828 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:03.597829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:03.597829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:03.597953 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:03.598037 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:03.598357 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 06:25:03.598449 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260926 06:25:03.599102 2452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:03.586961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36067 (local address 127.2.39.193:60891)
0926 06:25:03.587229 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:03.587278 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:03.587997 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:03.588222 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:03.588226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:03.588957 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:03.589225 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:03.589231 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.590040 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.590041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:03.591063 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:03.591066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.591539 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.591540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:03.591671 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:03.592231 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:03.592243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:03.594215 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:03.596736 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:03.596743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:03.596745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:03.597197 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:03.597539 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:03.597539 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:03.597540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:03.597765 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:03.598027 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:03.598034 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:03.598177 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 06:25:03.598685 (+ 508us) client_negotiation.cc:240] Negotiation successful
0926 06:25:03.598859 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":117,"threads_started":1}
I20260926 06:25:03.599763 2450 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36067
I20260926 06:25:03.599886 2450 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:03.600092 2450 heartbeater.cc:511] Master 127.2.39.254:36067 requested a full tablet report, sending...
I20260926 06:25:03.601086 2259 ts_manager.cc:194] Registered new tserver with Master: 4026a6600d6c4f17b9c9abdc2c5a4175 (127.2.39.193:43547)
I20260926 06:25:03.602100 2259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:60891
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:03.620738 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36067
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:03.716341 2455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:03.716555 2455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:03.716580 2455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:03.718119 2455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:03.718170 2455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:03.718199 2455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:03.720121 2455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:03.720353 2455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:03.720554 2455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:03.725752 2460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:03.725764 2461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:03.725845 2455 server_base.cc:1034] running on GCE node
W20260926 06:25:03.725764 2463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:03.726250 2455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:03.726509 2455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:03.727674 2455 hybrid_clock.cc:648] HybridClock initialized: now 1790403903727656 us; error 41 us; skew 500 ppm
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:03.730288 2455 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:03.730788 2455 webserver.cc:460] Webserver started at http://127.2.39.194:37461/ using document root <none> and password file <none>
I20260926 06:25:03.730968 2455 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:03.731016 2455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:03.731103 2455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:03.732218 2455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "7875db83822f4270aad49bd5d962a4cc"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "bc226712ec5e184bb08a7b964a79ab0a"
server_key_iv: "3f069b9a1f423cac43945475c65105ed"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.732573 2455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "7875db83822f4270aad49bd5d962a4cc"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "bc226712ec5e184bb08a7b964a79ab0a"
server_key_iv: "3f069b9a1f423cac43945475c65105ed"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.734097 2455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:03.734998 2470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:03.735219 2455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:03.735289 2455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "7875db83822f4270aad49bd5d962a4cc"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "bc226712ec5e184bb08a7b964a79ab0a"
server_key_iv: "3f069b9a1f423cac43945475c65105ed"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.735378 2455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:03.752079 2455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:03.754339 2455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:03.754457 2455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:03.754627 2455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:03.754945 2455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:03.755010 2455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:03.755074 2455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:03.755113 2455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:03.762190 2455 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:43253
I20260926 06:25:03.762240 2583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:43253 every 8 connection(s)
I20260926 06:25:03.762588 2455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:03.766012 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 2455
I20260926 06:25:03.766103 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:03.766260 2207 external_mini_cluster.cc:1428] Setting key 96084d38c67432619aa051bc60538120
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
I20260926 06:25:03.775347 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:03.763749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:51201 (local address 127.2.39.254:36067)
0926 06:25:03.763835 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:03.763837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:03.763957 (+ 120us) server_negotiation.cc:407] Connection header received
0926 06:25:03.764724 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:03.764725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:03.764738 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:03.764788 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:03.766225 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.767436 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.769684 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.769898 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.769986 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:03.772211 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:03.772213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:03.772213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:03.772221 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:03.773713 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:03.774236 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:03.774237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:03.774238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:03.774327 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:03.774637 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:03.774638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:03.774639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:03.774779 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:03.774857 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:03.775177 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 06:25:03.775255 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 06:25:03.775539 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:03.763689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36067 (local address 127.2.39.194:51201)
0926 06:25:03.763845 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:03.763886 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:03.764596 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:03.764860 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:03.764864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:03.765804 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:03.766111 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:03.766119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.768408 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.768409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:03.769610 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:03.769612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.770001 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.770002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:03.770146 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:03.770771 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:03.770779 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:03.772113 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:03.773805 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:03.773811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:03.773814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:03.774152 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:03.774421 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:03.774422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:03.774422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:03.774556 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:03.774845 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:03.774848 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:03.775017 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 06:25:03.775137 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 06:25:03.775292 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260926 06:25:03.776091 2584 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36067
I20260926 06:25:03.776196 2584 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:03.776392 2584 heartbeater.cc:511] Master 127.2.39.254:36067 requested a full tablet report, sending...
I20260926 06:25:03.776770 2259 ts_manager.cc:194] Registered new tserver with Master: 7875db83822f4270aad49bd5d962a4cc (127.2.39.194:43253)
I20260926 06:25:03.777155 2259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:51201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:03.793512 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36067
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:03.890390 2589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:03.890588 2589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:03.890650 2589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:03.892081 2589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:03.892124 2589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:03.892153 2589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:03.893983 2589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36067
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:03.894239 2589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:03.894481 2589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:03.899742 2597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:03.899890 2589 server_base.cc:1034] running on GCE node
W20260926 06:25:03.899742 2594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:03.899828 2595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:03.900156 2589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:03.900436 2589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:03.901647 2589 hybrid_clock.cc:648] HybridClock initialized: now 1790403903901637 us; error 39 us; skew 500 ppm
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:03.904170 2589 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:03.904640 2589 webserver.cc:460] Webserver started at http://127.2.39.195:42585/ using document root <none> and password file <none>
I20260926 06:25:03.904875 2589 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:03.904943 2589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:03.905078 2589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:03.906211 2589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "1372056253af463aad261d755a78073c"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "6d1faf331e0ac291933b5045deeec0fc"
server_key_iv: "5347839191e22cd5c0e24980be62679a"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.906574 2589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "1372056253af463aad261d755a78073c"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "6d1faf331e0ac291933b5045deeec0fc"
server_key_iv: "5347839191e22cd5c0e24980be62679a"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.907994 2589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:03.908763 2604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:03.908998 2589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:03.909094 2589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "1372056253af463aad261d755a78073c"
format_stamp: "Formatted at 2026-09-26 06:25:03 on dist-test-slave-fkbn"
server_key: "6d1faf331e0ac291933b5045deeec0fc"
server_key_iv: "5347839191e22cd5c0e24980be62679a"
server_key_version: "encryptionkey@0"
I20260926 06:25:03.909178 2589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:03.926544 2589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:03.928898 2589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:03.929061 2589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:03.929303 2589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:03.929754 2589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:03.929826 2589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:03.929858 2589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:03.929888 2589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:03.937695 2589 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:36449
I20260926 06:25:03.937739 2717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:36449 every 8 connection(s)
I20260926 06:25:03.938131 2589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:03.938588 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 2589
I20260926 06:25:03.938678 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:03.938932 2207 external_mini_cluster.cc:1428] Setting key 473585193420e8bbb9117a6ff4c4ead6
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:03.949410 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:03.939074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:44243 (local address 127.2.39.254:36067)
0926 06:25:03.939189 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:03.939191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:03.939564 (+ 373us) server_negotiation.cc:407] Connection header received
0926 06:25:03.940189 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:03.940190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:03.940204 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:03.940253 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:03.941661 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.942144 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.943608 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.943813 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.943865 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:03.946266 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:03.946267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:03.946267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:03.946274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:03.947871 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:03.948373 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:03.948374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:03.948374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:03.948428 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:03.948713 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:03.948714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:03.948714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:03.948814 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:03.948866 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:03.949279 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 06:25:03.949345 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 06:25:03.949934 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:03.939028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36067 (local address 127.2.39.195:44243)
0926 06:25:03.939295 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:03.939361 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:03.940077 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:03.940345 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:03.940350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:03.941195 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:03.941524 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:03.941530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.942274 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.942275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:03.943467 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:03.943469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.943912 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.943913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:03.944052 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:03.944520 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:03.944529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:03.946135 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:03.947974 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:03.947979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:03.947980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:03.948283 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:03.948525 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:03.948527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:03.948527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:03.948646 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:03.948881 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:03.948887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:03.949043 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 06:25:03.949168 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 06:25:03.949758 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
I20260926 06:25:03.950456 2718 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36067
I20260926 06:25:03.950573 2718 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:03.950765 2718 heartbeater.cc:511] Master 127.2.39.254:36067 requested a full tablet report, sending...
I20260926 06:25:03.951161 2259 ts_manager.cc:194] Registered new tserver with Master: 1372056253af463aad261d755a78073c (127.2.39.195:36449)
I20260926 06:25:03.951502 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:25:03.951567 2259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:44243
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:25:03.976347 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:03.967514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.2.39.193:43547)
0926 06:25:03.967736 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:03.967743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:03.967756 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:25:03.967789 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:03.967796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:03.967810 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:03.967892 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:03.968471 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.969480 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.970238 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.970689 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.970751 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:03.973114 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:03.973121 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:03.973121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:03.973127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:03.975071 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:03.975441 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:03.975444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:03.975445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:03.975544 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:03.975733 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:03.975734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:03.975734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:03.975926 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:03.976052 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:03.976125 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 06:25:03.976229 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
W20260926 06:25:03.976794 2364 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33922
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:25:03 dist-test-slave-fkbn krb5kdc[2214](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:25:04.001497 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:03.993019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.2.39.193:43547)
0926 06:25:03.993113 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:03.993115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:03.993148 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:25:03.993214 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:03.993215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:03.993224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:03.993275 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:03.994127 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.994982 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.995989 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:03.996245 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:03.996312 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:03.998909 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:03.998911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:03.998912 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:03.998919 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:04.000379 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:04.000703 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:04.000703 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:04.000704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:04.000771 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:04.000999 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:04.001000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:04.001000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:04.001098 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:04.001208 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:04.001270 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 06:25:04.001368 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:25:04.002111 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 2321
I20260926 06:25:04.007048 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 2455
I20260926 06:25:04.012758 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 2589
I20260926 06:25:04.017457 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 2227
2026-09-26T06:25:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4665 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:04 dist-test-slave-fkbn krb5kdc[2732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:04 dist-test-slave-fkbn krb5kdc[2732](info): set up 2 sockets
Sep 26 06:25:04 dist-test-slave-fkbn krb5kdc[2732](info): commencing operation
krb5kdc: starting...
W20260926 06:25:05.169427 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:05 dist-test-slave-fkbn krb5kdc[2732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:05.292773 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36645
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:36645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:05.383116 2745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:05.383317 2745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:05.383347 2745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:05.384756 2745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:05.384806 2745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:05.384824 2745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:05.384845 2745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:05.384869 2745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:05.386781 2745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:36645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:05.387064 2745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:05.387287 2745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:05.392438 2750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:05.392449 2753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:05.392445 2751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:05.392695 2745 server_base.cc:1034] running on GCE node
I20260926 06:25:05.392954 2745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:05.393299 2745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:05.394496 2745 hybrid_clock.cc:648] HybridClock initialized: now 1790403905394469 us; error 39 us; skew 500 ppm
Sep 26 06:25:05 dist-test-slave-fkbn krb5kdc[2732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403905, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:05.397332 2745 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:05.397816 2745 webserver.cc:460] Webserver started at http://127.2.39.254:44047/ using document root <none> and password file <none>
I20260926 06:25:05.398096 2745 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:05.398172 2745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:05.398337 2745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:05.399421 2745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "14d0420852b441caa4d985effcae2846"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "65495217ba1180f06ed3c44caeb1e53d"
server_key_iv: "ad4908fc9b001d83ad7e4578d3fb793e"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.399783 2745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "14d0420852b441caa4d985effcae2846"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "65495217ba1180f06ed3c44caeb1e53d"
server_key_iv: "ad4908fc9b001d83ad7e4578d3fb793e"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.401160 2745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:05.402209 2760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:05.402482 2745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:25:05.402566 2745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "14d0420852b441caa4d985effcae2846"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "65495217ba1180f06ed3c44caeb1e53d"
server_key_iv: "ad4908fc9b001d83ad7e4578d3fb793e"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.402621 2745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:05.422762 2745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:05.425493 2745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:05.425741 2745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:05.431041 2745 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:36645
I20260926 06:25:05.431116 2812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:36645 every 8 connection(s)
I20260926 06:25:05.431470 2745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:05.432337 2813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:05.434713 2813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846: Bootstrap starting.
I20260926 06:25:05.435310 2813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:05.435602 2813 log.cc:824] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:05.436374 2813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846: No bootstrap required, opened a new log
I20260926 06:25:05.437829 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 2745
I20260926 06:25:05.437923 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:05.438098 2207 external_mini_cluster.cc:1428] Setting key 4f63783d903baada44f9ee66849bcf17
I20260926 06:25:05.438220 2813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0420852b441caa4d985effcae2846" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36645 } }
I20260926 06:25:05.438416 2813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:05.438470 2813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d0420852b441caa4d985effcae2846, State: Initialized, Role: FOLLOWER
I20260926 06:25:05.438625 2813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0420852b441caa4d985effcae2846" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36645 } }
I20260926 06:25:05.438704 2813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:05.438738 2813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:05.438778 2813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:05.439983 2813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0420852b441caa4d985effcae2846" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36645 } }
I20260926 06:25:05.440107 2813 leader_election.cc:302] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14d0420852b441caa4d985effcae2846; no voters:
I20260926 06:25:05.440310 2813 leader_election.cc:288] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:05.440326 2818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:05.440511 2818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [term 1 LEADER]: Becoming Leader. State: Replica: 14d0420852b441caa4d985effcae2846, State: Running, Role: LEADER
I20260926 06:25:05.440573 2813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:05.440586 2818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0420852b441caa4d985effcae2846" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36645 } }
I20260926 06:25:05.441104 2820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14d0420852b441caa4d985effcae2846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0420852b441caa4d985effcae2846" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36645 } } }
I20260926 06:25:05.441218 2820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:05.441402 2819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d0420852b441caa4d985effcae2846. Latest consensus state: current_term: 1 leader_uuid: "14d0420852b441caa4d985effcae2846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14d0420852b441caa4d985effcae2846" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36645 } } }
I20260926 06:25:05.441470 2819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:05.441409 2827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:05.442071 2827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:05.443614 2827 catalog_manager.cc:1347] Generated new cluster ID: 17a9bb7a6ac446d6b0d490dd03211d35
I20260926 06:25:05.443657 2827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:25:05 dist-test-slave-fkbn krb5kdc[2732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:05.449525 2827 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:05.450251 2827 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:05.450925 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:05.438712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.2.39.254:36645)
0926 06:25:05.439148 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:05.439158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:05.439191 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:25:05.439323 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:05.439333 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:05.439402 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:05.439689 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:05.440548 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.442081 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.443137 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.443794 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.443896 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:05.446824 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:05.446828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:05.446828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:05.446841 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:05.448945 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:05.449504 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:05.449507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:05.449508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:05.449644 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:05.449863 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:05.449864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:05.449864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:05.450144 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:05.450298 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:05.450569 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 06:25:05.450730 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":76,"threads_started":1}
I20260926 06:25:05.456946 2827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14d0420852b441caa4d985effcae2846: Generated new TSK 0
I20260926 06:25:05.457123 2827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:05.484578 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36645
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:05.572500 2839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:05.572731 2839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:05.572772 2839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:05.574383 2839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:05.574476 2839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:05.574530 2839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:05.576119 2839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36645
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:05.576408 2839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:05.576665 2839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:05.582775 2845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:05.582769 2847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:05.583060 2844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:05.583084 2839 server_base.cc:1034] running on GCE node
I20260926 06:25:05.583321 2839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:05.583587 2839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:05.584780 2839 hybrid_clock.cc:648] HybridClock initialized: now 1790403905584761 us; error 32 us; skew 500 ppm
Sep 26 06:25:05 dist-test-slave-fkbn krb5kdc[2732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403905, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:05.587988 2839 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:05.588476 2839 webserver.cc:460] Webserver started at http://127.2.39.193:38505/ using document root <none> and password file <none>
I20260926 06:25:05.588749 2839 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:05.588794 2839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:05.588967 2839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:05.590312 2839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "8206d0651d354c00b498149c63a6470b"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "7f2d0a1e696f7ac5dfa70b68cbf867ee"
server_key_iv: "1cd9471b1c3c769462850b621ca41971"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.590802 2839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "8206d0651d354c00b498149c63a6470b"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "7f2d0a1e696f7ac5dfa70b68cbf867ee"
server_key_iv: "1cd9471b1c3c769462850b621ca41971"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.592780 2839 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 06:25:05.593739 2854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:05.593945 2839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:05.594072 2839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "8206d0651d354c00b498149c63a6470b"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "7f2d0a1e696f7ac5dfa70b68cbf867ee"
server_key_iv: "1cd9471b1c3c769462850b621ca41971"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.594156 2839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:05.606621 2839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:05.609163 2839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:05.609346 2839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:05.609637 2839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:05.609956 2839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:05.610010 2839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:05.610051 2839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:05.610078 2839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:05.616571 2839 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:46101
I20260926 06:25:05.616648 2967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:46101 every 8 connection(s)
I20260926 06:25:05.616971 2839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:05.619771 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 2839
I20260926 06:25:05.619916 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:05.620108 2207 external_mini_cluster.cc:1428] Setting key 55072034434550eff58d2142e1d24dc4
Sep 26 06:25:05 dist-test-slave-fkbn krb5kdc[2732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403905, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:05.629170 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:05.617709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:49519 (local address 127.2.39.254:36645)
0926 06:25:05.617764 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:05.617766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:05.618045 (+ 279us) server_negotiation.cc:407] Connection header received
0926 06:25:05.619190 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:05.619191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:05.619201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:05.619257 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:05.620032 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.620708 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.622437 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.622787 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.622886 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:05.625850 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:05.625852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:05.625852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:05.625860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:05.627554 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:05.628091 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:05.628092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:05.628093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:05.628153 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:05.628457 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:05.628457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:05.628458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:05.628565 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:05.628656 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:05.629025 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 06:25:05.629108 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
I20260926 06:25:05.629400 2970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:05.617716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36645 (local address 127.2.39.193:49519)
0926 06:25:05.617896 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:05.617943 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:05.619020 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:05.619325 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:05.619328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:05.619627 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:05.619846 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:05.619851 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.620875 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.620876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:05.622314 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:05.622318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.622921 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.622923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:05.623079 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:05.623827 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:05.623837 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:05.625740 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:05.627636 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:05.627642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:05.627645 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:05.628005 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:05.628223 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:05.628224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:05.628225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:05.628380 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:05.628649 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:05.628656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:05.628831 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 06:25:05.628990 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 06:25:05.629156 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260926 06:25:05.629938 2968 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36645
I20260926 06:25:05.630041 2968 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:05.630225 2968 heartbeater.cc:511] Master 127.2.39.254:36645 requested a full tablet report, sending...
I20260926 06:25:05.630759 2777 ts_manager.cc:194] Registered new tserver with Master: 8206d0651d354c00b498149c63a6470b (127.2.39.193:46101)
I20260926 06:25:05.631350 2777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:49519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:05.648108 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36645
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:05.732951 2973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:05.733152 2973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:05.733177 2973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:05.734623 2973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:05.734688 2973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:05.734719 2973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:05.736804 2973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36645
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:05.737069 2973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:05.737275 2973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:05.743503 2981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:05.743515 2979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:05.743492 2978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:05.743760 2973 server_base.cc:1034] running on GCE node
I20260926 06:25:05.743993 2973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:05.744283 2973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:05.745507 2973 hybrid_clock.cc:648] HybridClock initialized: now 1790403905745489 us; error 27 us; skew 500 ppm
Sep 26 06:25:05 dist-test-slave-fkbn krb5kdc[2732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403905, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:05.748375 2973 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:05.748904 2973 webserver.cc:460] Webserver started at http://127.2.39.194:46459/ using document root <none> and password file <none>
I20260926 06:25:05.749177 2973 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:05.749250 2973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:05.749389 2973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:05.750640 2973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "29dcdbf22d0d4fffbdbce1b0efae9844"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "5eb9e00a1ec799eb660a7027b91ea30d"
server_key_iv: "d32340459511c6d5da51734a790c540b"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.751042 2973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "29dcdbf22d0d4fffbdbce1b0efae9844"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "5eb9e00a1ec799eb660a7027b91ea30d"
server_key_iv: "d32340459511c6d5da51734a790c540b"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.752547 2973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:05.753366 2988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:05.753615 2973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:05.753712 2973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "29dcdbf22d0d4fffbdbce1b0efae9844"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "5eb9e00a1ec799eb660a7027b91ea30d"
server_key_iv: "d32340459511c6d5da51734a790c540b"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.753777 2973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:05.765931 2973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:05.768254 2973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:05.768391 2973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:05.768622 2973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:05.768922 2973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:05.768958 2973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:05.768980 2973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:05.769016 2973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:05.776229 2973 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:37527
I20260926 06:25:05.776269 3101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:37527 every 8 connection(s)
I20260926 06:25:05.776598 2973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:05.782675 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 2973
I20260926 06:25:05.782781 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:05.783002 2207 external_mini_cluster.cc:1428] Setting key 7493ca2034edb3c14c205a0d93348927
Sep 26 06:25:05 dist-test-slave-fkbn krb5kdc[2732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403905, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:05.787770 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:05.777329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:56337 (local address 127.2.39.254:36645)
0926 06:25:05.777370 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:05.777372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:05.777631 (+ 259us) server_negotiation.cc:407] Connection header received
0926 06:25:05.779198 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:05.779199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:05.779218 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:05.779281 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:05.779914 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.780513 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.781762 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.781981 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.782040 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:05.784578 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:05.784580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:05.784580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:05.784588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:05.786349 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:05.786770 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:05.786771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:05.786772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:05.786834 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:05.787115 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:05.787116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:05.787117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:05.787234 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:05.787325 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:05.787578 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 06:25:05.787671 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:25:05.788126 3104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:05.777294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36645 (local address 127.2.39.194:56337)
0926 06:25:05.777450 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:05.777489 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:05.779054 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:05.779335 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:05.779339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:05.779621 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:05.779804 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:05.779808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.780608 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.780609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:05.781653 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:05.781656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.782069 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.782070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:05.782179 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:05.782801 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:05.782812 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:05.784484 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:05.786441 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:05.786447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:05.786450 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:05.786687 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:05.786930 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:05.786931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:05.786931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:05.787043 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:05.787301 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:05.787306 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:05.787452 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 06:25:05.787560 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 06:25:05.787717 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260926 06:25:05.788589 3102 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36645
I20260926 06:25:05.788691 3102 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:05.789007 3102 heartbeater.cc:511] Master 127.2.39.254:36645 requested a full tablet report, sending...
I20260926 06:25:05.789408 2777 ts_manager.cc:194] Registered new tserver with Master: 29dcdbf22d0d4fffbdbce1b0efae9844 (127.2.39.194:37527)
I20260926 06:25:05.789932 2777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:56337
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:05.808915 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36645
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:05.901978 3107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:05.902302 3107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:05.902341 3107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:05.903553 3107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:05.903615 3107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:05.903661 3107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:05.905344 3107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36645
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:05.905648 3107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:05.905876 3107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:05.911669 3112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:05.911656 3113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:05.911656 3115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:05.911836 3107 server_base.cc:1034] running on GCE node
I20260926 06:25:05.912194 3107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:05.912472 3107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:05.913650 3107 hybrid_clock.cc:648] HybridClock initialized: now 1790403905913638 us; error 30 us; skew 500 ppm
Sep 26 06:25:05 dist-test-slave-fkbn krb5kdc[2732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403905, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:05.916265 3107 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:05.916787 3107 webserver.cc:460] Webserver started at http://127.2.39.195:35105/ using document root <none> and password file <none>
I20260926 06:25:05.917025 3107 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:05.917092 3107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:05.917316 3107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:05.918547 3107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "dd184fe970854530a46190005007fd08"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "c8f7d4d85b4800a7a31619f36504b230"
server_key_iv: "c4573bb7b14a8c805e9555205db57fa5"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.918927 3107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "dd184fe970854530a46190005007fd08"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "c8f7d4d85b4800a7a31619f36504b230"
server_key_iv: "c4573bb7b14a8c805e9555205db57fa5"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.920311 3107 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:25:05.921095 3122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:05.921280 3107 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:05.921398 3107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "dd184fe970854530a46190005007fd08"
format_stamp: "Formatted at 2026-09-26 06:25:05 on dist-test-slave-fkbn"
server_key: "c8f7d4d85b4800a7a31619f36504b230"
server_key_iv: "c4573bb7b14a8c805e9555205db57fa5"
server_key_version: "encryptionkey@0"
I20260926 06:25:05.921504 3107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:05.961313 3107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:05.963824 3107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:05.963994 3107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:05.964273 3107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:05.964607 3107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:05.964663 3107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:05.964704 3107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:05.964773 3107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:05.972325 3107 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:43821
I20260926 06:25:05.972391 3235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:43821 every 8 connection(s)
I20260926 06:25:05.972735 3107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:05.974424 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 3107
I20260926 06:25:05.974550 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:05.974749 2207 external_mini_cluster.cc:1428] Setting key e2ddfef271622a8d893c33d94f2e981a
Sep 26 06:25:05 dist-test-slave-fkbn krb5kdc[2732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403905, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:05.983579 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:05.973506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:49451 (local address 127.2.39.254:36645)
0926 06:25:05.973606 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:05.973609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:05.973794 (+ 185us) server_negotiation.cc:407] Connection header received
0926 06:25:05.975106 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:05.975108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:05.975120 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:05.975190 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:05.976069 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.976676 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.977913 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.978169 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.978228 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:05.980686 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:05.980688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:05.980688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:05.980698 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:05.982195 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:05.982645 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:05.982646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:05.982647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:05.982713 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:05.982989 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:05.982990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:05.982991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:05.983111 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:05.983165 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:05.983438 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 06:25:05.983516 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 06:25:05.983752 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:05.973491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36645 (local address 127.2.39.195:49451)
0926 06:25:05.973661 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:05.973720 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:05.974953 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:05.975279 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:05.975283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:05.975749 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:05.975988 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:05.975992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.976760 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.976761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:05.977839 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:05.977841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.978243 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.978244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:05.978345 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:05.978939 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:05.978949 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:05.980442 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:05.982300 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:05.982305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:05.982307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:05.982543 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:05.982785 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:05.982786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:05.982786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:05.982881 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:05.983169 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:05.983173 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:05.983312 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 06:25:05.983394 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 06:25:05.983529 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260926 06:25:05.984262 3236 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36645
I20260926 06:25:05.984380 3236 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:05.984566 3236 heartbeater.cc:511] Master 127.2.39.254:36645 requested a full tablet report, sending...
I20260926 06:25:05.985003 2777 ts_manager.cc:194] Registered new tserver with Master: dd184fe970854530a46190005007fd08 (127.2.39.195:43821)
I20260926 06:25:05.985414 2777 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:49451
I20260926 06:25:05.986581 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:25:05.994386 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:05.988185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.2.39.254:36645)
0926 06:25:05.988217 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:05.988219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:05.988326 (+ 107us) server_negotiation.cc:407] Connection header received
0926 06:25:05.988391 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:05.988392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:05.988404 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:05.988469 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:05.989052 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.989633 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.990374 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:05.990621 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:05.990677 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:05.991988 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:05.991989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:05.991989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:05.991995 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:05.993337 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:05.993694 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:05.993695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:05.993695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:05.993759 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:05.993918 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:05.993918 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:05.993918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:05.994092 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:05.994189 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:05.994276 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 06:25:05.994330 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 06:25:05.996695 2777 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:05.998713 2777 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:06.005959 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.001289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:37527 (local address 127.0.0.1:51636)
0926 06:25:06.001491 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:06.001533 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:06.001643 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:06.002061 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:06.002064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:06.002066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:06.002140 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:06.002142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.003498 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.003499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:06.004845 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:06.004847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.005607 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.005608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:06.005698 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.005712 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:25:06.005815 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 06:25:06.005877 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":57,"thread_start_us":65,"threads_started":1}
I20260926 06:25:06.006006 3250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.001551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51636 (local address 127.2.39.194:37527)
0926 06:25:06.001891 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:06.001894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:06.001904 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:06.001939 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:06.001941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:06.001958 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:06.002219 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:06.002298 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.003403 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.004924 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.005431 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.005537 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.005641 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:06.005864 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 06:25:06.005922 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":99,"threads_started":1}
I20260926 06:25:06.006495 3249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.001360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:46101 (local address 127.0.0.1:46370)
0926 06:25:06.001740 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:06.001750 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:06.001789 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:06.002153 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:06.002153 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:06.002156 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:06.002204 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:06.002206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.003425 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.003425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:06.004720 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:06.004722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.006181 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.006182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:06.006321 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.006325 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:06.006382 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 06:25:06.006422 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":36,"thread_start_us":81,"threads_started":1}
I20260926 06:25:06.006506 3252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.001726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.2.39.195:43821)
0926 06:25:06.001872 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:06.001877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:06.001945 (+ 68us) server_negotiation.cc:407] Connection header received
0926 06:25:06.002050 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:06.002052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:06.002066 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:06.002136 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:06.002402 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.003488 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.004889 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.005679 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.005749 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.005804 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:06.006383 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 06:25:06.006434 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260926 06:25:06.006564 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.001548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.2.39.193:46101)
0926 06:25:06.001904 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:06.001908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:06.001919 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:25:06.001960 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:06.001962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:06.001977 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:06.002054 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:06.002288 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.003273 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.004907 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.006023 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.006088 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.006150 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:06.006402 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 06:25:06.006467 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":206,"threads_started":1}
I20260926 06:25:06.006752 3036 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb29942b444756bcbaac3d7829ecd2 (DEFAULT_TABLE table=test-table [id=d4f888f349e54242adb8bfe5302b884f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:06.007056 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.001508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:43821 (local address 127.0.0.1:53618)
0926 06:25:06.001850 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:06.001862 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:06.001959 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:06.002248 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:06.002249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:06.002251 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:06.002292 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:06.002294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.003566 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.003567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:06.004801 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:06.004804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.005846 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.005847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:06.005918 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.005924 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:25:06.005982 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 06:25:06.006971 (+ 989us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":53,"thread_start_us":95,"threads_started":1}
I20260926 06:25:06.007143 3036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb29942b444756bcbaac3d7829ecd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:06.007793 3170 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb29942b444756bcbaac3d7829ecd2 (DEFAULT_TABLE table=test-table [id=d4f888f349e54242adb8bfe5302b884f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:06.008118 3170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb29942b444756bcbaac3d7829ecd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:06.010303 3253 tablet_bootstrap.cc:492] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844: Bootstrap starting.
I20260926 06:25:06.010275 2902 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb29942b444756bcbaac3d7829ecd2 (DEFAULT_TABLE table=test-table [id=d4f888f349e54242adb8bfe5302b884f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:06.010589 2902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb29942b444756bcbaac3d7829ecd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:06.011210 3253 tablet_bootstrap.cc:654] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:06.011317 3254 tablet_bootstrap.cc:492] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08: Bootstrap starting.
I20260926 06:25:06.011843 3253 log.cc:824] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:06.012118 3254 tablet_bootstrap.cc:654] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:06.012734 3253 tablet_bootstrap.cc:492] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844: No bootstrap required, opened a new log
I20260926 06:25:06.012735 3254 log.cc:824] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:06.012805 3253 ts_tablet_manager.cc:1397] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:25:06.014287 3254 tablet_bootstrap.cc:492] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08: No bootstrap required, opened a new log
I20260926 06:25:06.014438 3257 tablet_bootstrap.cc:492] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b: Bootstrap starting.
I20260926 06:25:06.014374 3254 ts_tablet_manager.cc:1397] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:25:06.015064 3257 tablet_bootstrap.cc:654] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:06.015269 3253 raft_consensus.cc:348] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 37527 } } peers { permanent_uuid: "8206d0651d354c00b498149c63a6470b" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46101 } } peers { permanent_uuid: "dd184fe970854530a46190005007fd08" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43821 } }
I20260926 06:25:06.015473 3253 raft_consensus.cc:374] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:06.015525 3253 raft_consensus.cc:729] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29dcdbf22d0d4fffbdbce1b0efae9844, State: Initialized, Role: FOLLOWER
I20260926 06:25:06.015599 3257 log.cc:824] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:06.015692 3253 consensus_queue.cc:260] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 37527 } } peers { permanent_uuid: "8206d0651d354c00b498149c63a6470b" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46101 } } peers { permanent_uuid: "dd184fe970854530a46190005007fd08" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43821 } }
I20260926 06:25:06.016005 3253 ts_tablet_manager.cc:1428] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 06:25:06.016099 3102 heartbeater.cc:503] Master 127.2.39.254:36645 was elected leader, sending a full tablet report...
I20260926 06:25:06.016675 3257 tablet_bootstrap.cc:492] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b: No bootstrap required, opened a new log
I20260926 06:25:06.016745 3257 ts_tablet_manager.cc:1397] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:25:06.016781 3254 raft_consensus.cc:348] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 37527 } } peers { permanent_uuid: "8206d0651d354c00b498149c63a6470b" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46101 } } peers { permanent_uuid: "dd184fe970854530a46190005007fd08" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43821 } }
I20260926 06:25:06.016978 3254 raft_consensus.cc:374] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:06.017019 3254 raft_consensus.cc:729] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd184fe970854530a46190005007fd08, State: Initialized, Role: FOLLOWER
I20260926 06:25:06.017143 3254 consensus_queue.cc:260] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 37527 } } peers { permanent_uuid: "8206d0651d354c00b498149c63a6470b" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46101 } } peers { permanent_uuid: "dd184fe970854530a46190005007fd08" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43821 } }
I20260926 06:25:06.017409 3254 ts_tablet_manager.cc:1428] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:25:06.017431 3236 heartbeater.cc:503] Master 127.2.39.254:36645 was elected leader, sending a full tablet report...
I20260926 06:25:06.018697 3257 raft_consensus.cc:348] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 37527 } } peers { permanent_uuid: "8206d0651d354c00b498149c63a6470b" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46101 } } peers { permanent_uuid: "dd184fe970854530a46190005007fd08" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43821 } }
I20260926 06:25:06.018848 3257 raft_consensus.cc:374] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:06.018981 3257 raft_consensus.cc:729] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8206d0651d354c00b498149c63a6470b, State: Initialized, Role: FOLLOWER
I20260926 06:25:06.019106 3257 consensus_queue.cc:260] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 37527 } } peers { permanent_uuid: "8206d0651d354c00b498149c63a6470b" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46101 } } peers { permanent_uuid: "dd184fe970854530a46190005007fd08" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43821 } }
I20260926 06:25:06.019384 3257 ts_tablet_manager.cc:1428] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:25:06.019389 2968 heartbeater.cc:503] Master 127.2.39.254:36645 was elected leader, sending a full tablet report...
W20260926 06:25:06.027137 3103 tablet.cc:2369] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:25:06.117763 2969 tablet.cc:2369] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:25:06.136686 3261 raft_consensus.cc:482] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:06.136932 3261 raft_consensus.cc:504] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 37527 } } peers { permanent_uuid: "8206d0651d354c00b498149c63a6470b" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46101 } } peers { permanent_uuid: "dd184fe970854530a46190005007fd08" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43821 } }
I20260926 06:25:06.137419 3261 leader_election.cc:288] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29dcdbf22d0d4fffbdbce1b0efae9844 (127.2.39.194:37527), dd184fe970854530a46190005007fd08 (127.2.39.195:43821)
I20260926 06:25:06.141206 3252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.137944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:42671 (local address 127.2.39.195:43821)
0926 06:25:06.137991 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:06.137993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:06.138066 (+ 73us) server_negotiation.cc:407] Connection header received
0926 06:25:06.138130 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:06.138131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:06.138150 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:06.138229 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:06.138391 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.138858 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.140268 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.140885 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.140955 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.140964 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:06.141102 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 06:25:06.141142 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:25:06.141225 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.137849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:43821 (local address 127.2.39.193:42671)
0926 06:25:06.137979 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:06.137988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:06.138077 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:06.138230 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:06.138231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:06.138232 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:06.138316 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:06.138317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.138938 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.138939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:06.140199 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:06.140200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.140966 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.140967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:06.141021 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.141025 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:06.141097 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 06:25:06.141139 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":14,"thread_start_us":44,"threads_started":1}
I20260926 06:25:06.141619 3190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb29942b444756bcbaac3d7829ecd2" candidate_uuid: "8206d0651d354c00b498149c63a6470b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd184fe970854530a46190005007fd08" is_pre_election: true
I20260926 06:25:06.141786 3190 raft_consensus.cc:2454] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8206d0651d354c00b498149c63a6470b in term 0.
I20260926 06:25:06.142024 2856 leader_election.cc:302] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8206d0651d354c00b498149c63a6470b, dd184fe970854530a46190005007fd08; no voters:
I20260926 06:25:06.142088 3250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.137763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:46931 (local address 127.2.39.194:37527)
0926 06:25:06.137846 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:06.137849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:06.137863 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:25:06.137902 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:06.137903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:06.137916 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:06.138002 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:06.138231 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.138952 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.141022 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.141655 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.141735 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.141745 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:06.141977 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 06:25:06.142019 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:25:06.142134 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.137657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:37527 (local address 127.2.39.193:46931)
0926 06:25:06.137780 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:06.137792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:06.137847 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:06.138049 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:06.138050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:06.138053 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:06.138129 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:06.138132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.139054 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.139054 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:06.140192 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:06.140194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.141797 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.141798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:06.141904 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.141910 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:25:06.141969 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:25:06.142011 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260926 06:25:06.142171 3261 raft_consensus.cc:2781] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:06.142217 3261 raft_consensus.cc:482] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:06.142243 3261 raft_consensus.cc:3037] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:06.142434 3056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb29942b444756bcbaac3d7829ecd2" candidate_uuid: "8206d0651d354c00b498149c63a6470b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" is_pre_election: true
I20260926 06:25:06.142602 3056 raft_consensus.cc:2454] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8206d0651d354c00b498149c63a6470b in term 0.
I20260926 06:25:06.143072 3261 raft_consensus.cc:504] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 37527 } } peers { permanent_uuid: "8206d0651d354c00b498149c63a6470b" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46101 } } peers { permanent_uuid: "dd184fe970854530a46190005007fd08" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43821 } }
I20260926 06:25:06.143174 3261 leader_election.cc:288] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [CANDIDATE]: Term 1 election: Requested vote from peers 29dcdbf22d0d4fffbdbce1b0efae9844 (127.2.39.194:37527), dd184fe970854530a46190005007fd08 (127.2.39.195:43821)
I20260926 06:25:06.143473 3056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb29942b444756bcbaac3d7829ecd2" candidate_uuid: "8206d0651d354c00b498149c63a6470b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844"
I20260926 06:25:06.143500 3190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb29942b444756bcbaac3d7829ecd2" candidate_uuid: "8206d0651d354c00b498149c63a6470b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd184fe970854530a46190005007fd08"
I20260926 06:25:06.143561 3190 raft_consensus.cc:3037] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:06.143553 3056 raft_consensus.cc:3037] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:06.144428 3056 raft_consensus.cc:2454] T f2cb29942b444756bcbaac3d7829ecd2 P 29dcdbf22d0d4fffbdbce1b0efae9844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8206d0651d354c00b498149c63a6470b in term 1.
I20260926 06:25:06.144500 3190 raft_consensus.cc:2454] T f2cb29942b444756bcbaac3d7829ecd2 P dd184fe970854530a46190005007fd08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8206d0651d354c00b498149c63a6470b in term 1.
I20260926 06:25:06.144629 2856 leader_election.cc:302] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29dcdbf22d0d4fffbdbce1b0efae9844, 8206d0651d354c00b498149c63a6470b; no voters:
I20260926 06:25:06.144713 3261 raft_consensus.cc:2781] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:06.144858 3261 raft_consensus.cc:686] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [term 1 LEADER]: Becoming Leader. State: Replica: 8206d0651d354c00b498149c63a6470b, State: Running, Role: LEADER
I20260926 06:25:06.144932 3261 consensus_queue.cc:237] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 37527 } } peers { permanent_uuid: "8206d0651d354c00b498149c63a6470b" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46101 } } peers { permanent_uuid: "dd184fe970854530a46190005007fd08" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43821 } }
I20260926 06:25:06.145859 2777 catalog_manager.cc:5495] T f2cb29942b444756bcbaac3d7829ecd2 P 8206d0651d354c00b498149c63a6470b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8206d0651d354c00b498149c63a6470b (127.2.39.193). New cstate: current_term: 1 leader_uuid: "8206d0651d354c00b498149c63a6470b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29dcdbf22d0d4fffbdbce1b0efae9844" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 37527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8206d0651d354c00b498149c63a6470b" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd184fe970854530a46190005007fd08" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43821 } health_report { overall_health: UNKNOWN } } }
Sep 26 06:25:06 dist-test-slave-fkbn krb5kdc[2732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403905, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:25:06.156077 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.148118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.2.39.193:46101)
0926 06:25:06.148175 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:06.148177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:06.148326 (+ 149us) server_negotiation.cc:407] Connection header received
0926 06:25:06.148397 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:06.148398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:06.148409 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:06.148458 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:06.149014 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.149520 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.150273 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.150478 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.150555 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.153081 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:06.153089 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:06.153090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:06.153096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:06.154870 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:06.155281 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:06.155285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:06.155285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:06.155357 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:06.155556 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:06.155557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:06.155558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:06.155689 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:06.155842 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:06.155892 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 06:25:06.155979 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 06:25:06 dist-test-slave-fkbn krb5kdc[2732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403906, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:25:06 dist-test-slave-fkbn krb5kdc[2732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403906, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:25:06.182497 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.174678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.2.39.193:46101)
0926 06:25:06.174746 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:06.174748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:06.174849 (+ 101us) server_negotiation.cc:407] Connection header received
0926 06:25:06.174933 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:06.174934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:06.174944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:06.175024 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:06.175720 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.176194 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.176973 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.177167 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.177267 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.179755 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:06.179756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:06.179757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:06.179765 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:06.181316 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:06.181665 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:06.181666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:06.181666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:06.181715 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:06.181891 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:06.181892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:06.181893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:06.182010 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:06.182103 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:06.182239 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 06:25:06.182346 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 06:25:06 dist-test-slave-fkbn krb5kdc[2732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:25:06 dist-test-slave-fkbn krb5kdc[2732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:25:06.206837 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:06.198799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.2.39.193:46101)
0926 06:25:06.198880 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:06.198882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:06.199010 (+ 128us) server_negotiation.cc:407] Connection header received
0926 06:25:06.199120 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:06.199121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:06.199133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:06.199198 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:06.199968 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.200532 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.201402 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:06.201689 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:06.201747 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:06.204194 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:06.204195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:06.204196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:06.204203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:06.205700 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:06.206053 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:06.206055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:06.206055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:06.206110 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:06.206305 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:06.206306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:06.206306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:06.206403 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:06.206526 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:06.206653 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 06:25:06.206733 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 06:25:06.207365 2882 tablet_service.cc:2908] Rejecting scan request for tablet f2cb29942b444756bcbaac3d7829ecd2: Uninitialized: clean time has not yet been initialized
I20260926 06:25:06.208194 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 2839
I20260926 06:25:06.213348 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 2973
I20260926 06:25:06.218307 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 3107
I20260926 06:25:06.223063 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 2745
2026-09-26T06:25:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2206 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:06 dist-test-slave-fkbn krb5kdc[3278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:06 dist-test-slave-fkbn krb5kdc[3278](info): set up 2 sockets
Sep 26 06:25:06 dist-test-slave-fkbn krb5kdc[3278](info): commencing operation
krb5kdc: starting...
W20260926 06:25:07.334635 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:07 dist-test-slave-fkbn krb5kdc[3278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:07.484841 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38215
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:07.592316 3291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:07.592592 3291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:07.592633 3291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:07.594538 3291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:07.594590 3291 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 06:25:07.594611 3291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:07.594643 3291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:07.594661 3291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:07.594686 3291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:07.596794 3291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:38215
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:07.597069 3291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:07.597343 3291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:07.603591 3297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:07.603756 3291 server_base.cc:1034] running on GCE node
W20260926 06:25:07.603591 3296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:07.603683 3299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:07.604086 3291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:07.604321 3291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:07.605444 3291 hybrid_clock.cc:648] HybridClock initialized: now 1790403907605431 us; error 30 us; skew 500 ppm
Sep 26 06:25:07 dist-test-slave-fkbn krb5kdc[3278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403907, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:07.608376 3291 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:07.608922 3291 webserver.cc:460] Webserver started at http://127.2.39.254:34491/ using document root <none> and password file <none>
I20260926 06:25:07.609210 3291 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:07.609280 3291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:07.609431 3291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:07.610962 3291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "e4510fb0b0e740f0855362bd27eb24bb"
format_stamp: "Formatted at 2026-09-26 06:25:07 on dist-test-slave-fkbn"
server_key: "4764001f3ef0dbb387b19511d8e4f962"
server_key_iv: "4ac8e46e251c854de4d7c5e9124d9de7"
server_key_version: "encryptionkey@0"
I20260926 06:25:07.611383 3291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "e4510fb0b0e740f0855362bd27eb24bb"
format_stamp: "Formatted at 2026-09-26 06:25:07 on dist-test-slave-fkbn"
server_key: "4764001f3ef0dbb387b19511d8e4f962"
server_key_iv: "4ac8e46e251c854de4d7c5e9124d9de7"
server_key_version: "encryptionkey@0"
I20260926 06:25:07.613201 3291 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 06:25:07.614013 3306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:07.614219 3291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:25:07.614315 3291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "e4510fb0b0e740f0855362bd27eb24bb"
format_stamp: "Formatted at 2026-09-26 06:25:07 on dist-test-slave-fkbn"
server_key: "4764001f3ef0dbb387b19511d8e4f962"
server_key_iv: "4ac8e46e251c854de4d7c5e9124d9de7"
server_key_version: "encryptionkey@0"
I20260926 06:25:07.614395 3291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:07.637748 3291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:07.640157 3291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:07.640326 3291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:07.646001 3291 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:38215
I20260926 06:25:07.646062 3368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:38215 every 8 connection(s)
I20260926 06:25:07.646440 3291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:07.647251 3369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:07.650225 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 3291
I20260926 06:25:07.650318 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:07.650345 3369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb: Bootstrap starting.
I20260926 06:25:07.650497 2207 external_mini_cluster.cc:1428] Setting key 6d4e2a3514daf199ad9bbf3bf2ced348
I20260926 06:25:07.651057 3369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:07.651386 3369 log.cc:824] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:07.652133 3369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb: No bootstrap required, opened a new log
I20260926 06:25:07.654449 3369 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4510fb0b0e740f0855362bd27eb24bb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38215 } }
I20260926 06:25:07.654632 3369 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:07.654668 3369 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4510fb0b0e740f0855362bd27eb24bb, State: Initialized, Role: FOLLOWER
I20260926 06:25:07.654791 3369 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4510fb0b0e740f0855362bd27eb24bb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38215 } }
I20260926 06:25:07.654860 3369 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:07.654887 3369 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:07.654922 3369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:07.655632 3369 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4510fb0b0e740f0855362bd27eb24bb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38215 } }
I20260926 06:25:07.655738 3369 leader_election.cc:302] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4510fb0b0e740f0855362bd27eb24bb; no voters:
I20260926 06:25:07.655898 3369 leader_election.cc:288] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:07.655927 3374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:07.656045 3374 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [term 1 LEADER]: Becoming Leader. State: Replica: e4510fb0b0e740f0855362bd27eb24bb, State: Running, Role: LEADER
I20260926 06:25:07.656126 3374 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4510fb0b0e740f0855362bd27eb24bb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38215 } }
I20260926 06:25:07.656164 3369 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:07.656425 3374 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4510fb0b0e740f0855362bd27eb24bb. Latest consensus state: current_term: 1 leader_uuid: "e4510fb0b0e740f0855362bd27eb24bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4510fb0b0e740f0855362bd27eb24bb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38215 } } }
I20260926 06:25:07.656505 3374 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:07.656657 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4510fb0b0e740f0855362bd27eb24bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4510fb0b0e740f0855362bd27eb24bb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38215 } } }
I20260926 06:25:07.656749 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb [sys.catalog]: This master's current role is: LEADER
W20260926 06:25:07.657934 3389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:25:07.658030 3389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:25:07.658274 3379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 06:25:07 dist-test-slave-fkbn krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:07.658453 3379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:07.660162 3379 catalog_manager.cc:1347] Generated new cluster ID: d3f3053f5ea747ea81552cba803978ca
I20260926 06:25:07.660245 3379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:07.663023 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:07.650955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39514 (local address 127.2.39.254:38215)
0926 06:25:07.651161 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:07.651172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:07.651208 (+ 36us) server_negotiation.cc:407] Connection header received
0926 06:25:07.651793 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:07.651800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:07.651894 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:07.652043 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:07.652714 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:07.654246 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:07.655286 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:07.656027 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:07.656415 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:07.658964 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:07.658968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:07.658968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:07.658980 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:07.660966 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:07.661362 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:07.661365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:07.661365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:07.661462 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:07.661715 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:07.661717 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:07.661717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:07.662027 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:07.662180 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:07.662667 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 06:25:07.662799 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260926 06:25:07.663890 3379 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:07.664346 3379 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:07.669289 3379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb: Generated new TSK 0
I20260926 06:25:07.669453 3379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:07.700996 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38215
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:07.828058 3395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:07.828344 3395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:07.828384 3395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:07.830623 3395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:07.830686 3395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:07.830739 3395 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:25:07.830771 3395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:07.833412 3395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.39.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:07.833724 3395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:07.834012 3395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:07.839790 3401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:07.839824 3400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:07.840103 3395 server_base.cc:1034] running on GCE node
W20260926 06:25:07.840226 3403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:07.840473 3395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:07.840729 3395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:07.841887 3395 hybrid_clock.cc:648] HybridClock initialized: now 1790403907841879 us; error 45 us; skew 500 ppm
Sep 26 06:25:07 dist-test-slave-fkbn krb5kdc[3278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403907, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:07.845340 3395 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:07.845913 3395 webserver.cc:460] Webserver started at http://127.2.39.193:41885/ using document root <none> and password file <none>
I20260926 06:25:07.846205 3395 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:07.846276 3395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:07.846424 3395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:07.847944 3395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "a8988947728c4d9a96b7ea6fdd60ba07"
format_stamp: "Formatted at 2026-09-26 06:25:07 on dist-test-slave-fkbn"
server_key: "546a2a6aea93d99c435e8abfe9664543"
server_key_iv: "1b794e675bc4256946e46f5bba04d8cf"
server_key_version: "encryptionkey@0"
I20260926 06:25:07.848402 3395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "a8988947728c4d9a96b7ea6fdd60ba07"
format_stamp: "Formatted at 2026-09-26 06:25:07 on dist-test-slave-fkbn"
server_key: "546a2a6aea93d99c435e8abfe9664543"
server_key_iv: "1b794e675bc4256946e46f5bba04d8cf"
server_key_version: "encryptionkey@0"
I20260926 06:25:07.850270 3395 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260926 06:25:07.851334 3410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:07.851521 3395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:07.851619 3395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "a8988947728c4d9a96b7ea6fdd60ba07"
format_stamp: "Formatted at 2026-09-26 06:25:07 on dist-test-slave-fkbn"
server_key: "546a2a6aea93d99c435e8abfe9664543"
server_key_iv: "1b794e675bc4256946e46f5bba04d8cf"
server_key_version: "encryptionkey@0"
I20260926 06:25:07.851709 3395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:07.875757 3395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:07.878582 3395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:07.878751 3395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:07.879434 3395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:07.879487 3395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:07.879522 3395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:07.879561 3395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:07.886957 3395 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:42181
I20260926 06:25:07.887051 3525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:42181 every 8 connection(s)
I20260926 06:25:07.887379 3395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:25:07 dist-test-slave-fkbn krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403907, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:07.891556 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:07.879802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:53113 (local address 127.2.39.254:38215)
0926 06:25:07.879850 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:07.879852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:07.880111 (+ 259us) server_negotiation.cc:407] Connection header received
0926 06:25:07.880606 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:07.880607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:07.880620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:07.880700 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:07.881767 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:07.882623 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:07.884515 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:07.884861 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:07.884911 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:07.888351 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:07.888352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:07.888353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:07.888361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:07.889873 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:07.890374 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:07.890375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:07.890376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:07.890455 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:07.890761 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:07.890762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:07.890762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:07.890898 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:07.890982 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:07.891330 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 06:25:07.891435 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:25:07.891742 3419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:07.879813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38215 (local address 127.2.39.193:53113)
0926 06:25:07.879975 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:07.880017 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:07.880511 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:07.880847 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:07.880853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:07.881255 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:07.881629 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:07.881637 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:07.882759 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:07.882760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:07.884268 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:07.884271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:07.885016 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:07.885017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:07.885183 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:07.886382 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:07.886393 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:07.888239 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:07.889973 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:07.889979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:07.889982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:07.890301 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:07.890547 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:07.890548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:07.890549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:07.890692 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:07.891014 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:07.891020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:07.891182 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 06:25:07.891293 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 06:25:07.891483 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260926 06:25:07.892328 3526 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38215
I20260926 06:25:07.892416 3526 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:07.892611 3526 heartbeater.cc:511] Master 127.2.39.254:38215 requested a full tablet report, sending...
I20260926 06:25:07.893728 3323 ts_manager.cc:194] Registered new tserver with Master: a8988947728c4d9a96b7ea6fdd60ba07 (127.2.39.193:42181)
I20260926 06:25:07.894599 3323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:53113
I20260926 06:25:07.897336 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 3395
I20260926 06:25:07.897493 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:07.897696 2207 external_mini_cluster.cc:1428] Setting key 7e400040c0b9f3b66974a095c34c6f69
I20260926 06:25:07.901324 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:07.893526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:38949 (local address 127.2.39.254:38215)
0926 06:25:07.894821 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:07.894823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:07.894833 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:07.894852 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:07.894853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:07.894863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:07.894921 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:07.895296 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:07.895952 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:07.896955 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:07.897200 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:07.897266 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:07.898420 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:07.898421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:07.898421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:07.898428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:07.900161 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:07.900496 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:07.900497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:07.900497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:07.900566 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:07.900791 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:07.900792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:07.900792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:07.900944 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:07.900990 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:07.901187 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 06:25:07.901260 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1276}
I20260926 06:25:07.901479 3532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:07.893458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38215 (local address 127.2.39.193:38949)
0926 06:25:07.893618 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:07.893630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:07.893670 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:07.894940 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:07.894941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:07.895161 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:07.895216 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:07.895218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:07.896061 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:07.896062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:07.896851 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:07.896853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:07.897327 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:07.897328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:07.897406 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:07.897865 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:07.897880 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:07.898294 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:07.900245 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:07.900246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:07.900246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:07.900422 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:07.900632 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:07.900632 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:07.900632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:07.900686 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:07.901038 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:07.901038 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:07.901086 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 06:25:07.901152 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 06:25:07.901270 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:07.928779 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38215
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:08.048640 3535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:08.048908 3535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:08.048941 3535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:08.050890 3535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:08.050945 3535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:08.050976 3535 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:25:08.051002 3535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:08.053238 3535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.39.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:08.053521 3535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:08.053798 3535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:08.059360 3540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:08.059407 3541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:08.059623 3535 server_base.cc:1034] running on GCE node
W20260926 06:25:08.059783 3543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:08.060027 3535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:08.060302 3535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:08.061470 3535 hybrid_clock.cc:648] HybridClock initialized: now 1790403908061448 us; error 46 us; skew 500 ppm
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403908, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:08.064849 3535 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:08.065407 3535 webserver.cc:460] Webserver started at http://127.2.39.194:41677/ using document root <none> and password file <none>
I20260926 06:25:08.065744 3535 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:08.065806 3535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:08.065932 3535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:08.067265 3535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "25e03a97b15945709c80d501ff1b3135"
format_stamp: "Formatted at 2026-09-26 06:25:08 on dist-test-slave-fkbn"
server_key: "d02b01e659e0339b46d94895091c7009"
server_key_iv: "63bd373df1f91eb8c33201c9dfe86795"
server_key_version: "encryptionkey@0"
I20260926 06:25:08.067739 3535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "25e03a97b15945709c80d501ff1b3135"
format_stamp: "Formatted at 2026-09-26 06:25:08 on dist-test-slave-fkbn"
server_key: "d02b01e659e0339b46d94895091c7009"
server_key_iv: "63bd373df1f91eb8c33201c9dfe86795"
server_key_version: "encryptionkey@0"
I20260926 06:25:08.069516 3535 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 06:25:08.070427 3550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:08.070570 3535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:08.070652 3535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "25e03a97b15945709c80d501ff1b3135"
format_stamp: "Formatted at 2026-09-26 06:25:08 on dist-test-slave-fkbn"
server_key: "d02b01e659e0339b46d94895091c7009"
server_key_iv: "63bd373df1f91eb8c33201c9dfe86795"
server_key_version: "encryptionkey@0"
I20260926 06:25:08.070744 3535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:08.098083 3535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:08.100565 3535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:08.100680 3535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:08.101246 3535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:08.101291 3535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:08.101315 3535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:08.101326 3535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:08.108669 3535 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:46759
I20260926 06:25:08.108771 3665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:46759 every 8 connection(s)
I20260926 06:25:08.109063 3535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403908, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:08.114303 3559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.101796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38215 (local address 127.2.39.194:59285)
0926 06:25:08.101968 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.102032 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.102755 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.103101 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.103105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.103571 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:08.103928 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.103937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.105129 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.105130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.106668 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.106672 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.107310 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.107311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.107485 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.108323 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:08.108335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:08.110563 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:08.112602 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:08.112609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:08.112612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:08.112848 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:08.113072 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:08.113073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:08.113073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:08.113195 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:08.113464 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:08.113470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:08.113691 (+ 221us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.113810 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.114011 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260926 06:25:08.114653 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.101789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:59285 (local address 127.2.39.254:38215)
0926 06:25:08.101851 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.101854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.102131 (+ 277us) server_negotiation.cc:407] Connection header received
0926 06:25:08.102901 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.102903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.102917 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.102991 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:08.104109 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.105000 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.106809 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.107221 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.107307 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.110679 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:08.110681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:08.110681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:08.110692 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:08.112491 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.112929 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.112930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.112931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.113002 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.113271 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.113272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.113272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.113395 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:08.113472 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.114460 (+ 988us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.114549 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:25:08.115070 3666 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38215
I20260926 06:25:08.115196 3666 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:08.115386 3666 heartbeater.cc:511] Master 127.2.39.254:38215 requested a full tablet report, sending...
I20260926 06:25:08.115442 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 3535
I20260926 06:25:08.115545 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:08.115762 2207 external_mini_cluster.cc:1428] Setting key fa012bcc73ca19b16cf362bf23365a23
I20260926 06:25:08.116240 3323 ts_manager.cc:194] Registered new tserver with Master: 25e03a97b15945709c80d501ff1b3135 (127.2.39.194:46759)
I20260926 06:25:08.116847 3323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:59285
I20260926 06:25:08.123637 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.116298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38215 (local address 127.2.39.194:42135)
0926 06:25:08.116478 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.116495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.116578 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.116711 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.116712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.116930 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:08.117014 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.117016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.117846 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.117847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.118766 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.118770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.119315 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.119316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.119404 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.120236 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:08.120242 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:08.120608 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:08.122468 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:08.122468 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:08.122469 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:08.122642 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:08.122858 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:08.122859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:08.122859 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:08.122918 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:08.123203 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:08.123204 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:08.123250 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.123372 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.123503 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":65,"threads_started":1}
I20260926 06:25:08.123677 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.116376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:42135 (local address 127.2.39.254:38215)
0926 06:25:08.116415 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.116417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.116552 (+ 135us) server_negotiation.cc:407] Connection header received
0926 06:25:08.116641 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.116642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.116654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.116703 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:08.117117 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.117726 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.118870 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.119218 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.119302 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.120720 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:08.120721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:08.120722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:08.120729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:08.122355 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.122730 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.122731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.122732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.122785 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.122991 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.122992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.122993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.123134 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:08.123211 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.123425 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.123528 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:08.144405 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38215
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:08.229123 3675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:08.229302 3675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:08.229326 3675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:08.230504 3675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:08.230544 3675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:08.230566 3675 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:25:08.230582 3675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:08.231941 3675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.39.254:38215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:08.232177 3675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:08.232407 3675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:08.237612 3680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:08.237594 3683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:08.237551 3681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:08.237823 3675 server_base.cc:1034] running on GCE node
I20260926 06:25:08.238057 3675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:08.238296 3675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:08.239456 3675 hybrid_clock.cc:648] HybridClock initialized: now 1790403908239431 us; error 54 us; skew 500 ppm
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403908, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:08.242089 3675 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:08.242547 3675 webserver.cc:460] Webserver started at http://127.2.39.195:36733/ using document root <none> and password file <none>
I20260926 06:25:08.242763 3675 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:08.242830 3675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:08.242982 3675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:08.243978 3675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "3ce88f7f49cc4d98bd069f23c4449013"
format_stamp: "Formatted at 2026-09-26 06:25:08 on dist-test-slave-fkbn"
server_key: "730bff615ae970ee66459a199e46bdec"
server_key_iv: "28b16dcd26afe4411427ee957ce87aa6"
server_key_version: "encryptionkey@0"
I20260926 06:25:08.244325 3675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "3ce88f7f49cc4d98bd069f23c4449013"
format_stamp: "Formatted at 2026-09-26 06:25:08 on dist-test-slave-fkbn"
server_key: "730bff615ae970ee66459a199e46bdec"
server_key_iv: "28b16dcd26afe4411427ee957ce87aa6"
server_key_version: "encryptionkey@0"
I20260926 06:25:08.245683 3675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 06:25:08.246420 3690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:08.246572 3675 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:08.246651 3675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "3ce88f7f49cc4d98bd069f23c4449013"
format_stamp: "Formatted at 2026-09-26 06:25:08 on dist-test-slave-fkbn"
server_key: "730bff615ae970ee66459a199e46bdec"
server_key_iv: "28b16dcd26afe4411427ee957ce87aa6"
server_key_version: "encryptionkey@0"
I20260926 06:25:08.246702 3675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:08.267424 3675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:08.269182 3675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:08.269284 3675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:08.269826 3675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:08.269865 3675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:08.269886 3675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:08.269933 3675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403908, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:08.276851 3675 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:36545
I20260926 06:25:08.276916 3805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:36545 every 8 connection(s)
I20260926 06:25:08.277235 3675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:08.279627 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 3675
I20260926 06:25:08.279719 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:08.279883 2207 external_mini_cluster.cc:1428] Setting key 5921d54b70c35ac44c6fb033b46c97c6
I20260926 06:25:08.280448 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.270286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:57299 (local address 127.2.39.254:38215)
0926 06:25:08.270353 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.270355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.270534 (+ 179us) server_negotiation.cc:407] Connection header received
0926 06:25:08.270999 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.271001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.271013 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.271100 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:08.271744 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.272574 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.274150 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.274491 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.274570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.277355 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:08.277357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:08.277357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:08.277364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:08.278944 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.279299 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.279300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.279300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.279348 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.279565 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.279565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.279566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.279712 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:08.279785 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.280174 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.280269 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:25:08.280529 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.270254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38215 (local address 127.2.39.195:57299)
0926 06:25:08.270407 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.270446 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.270908 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.271149 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.271152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.271383 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:08.271643 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.271648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.272675 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.272675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.274041 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.274045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.274593 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.274594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.274756 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.275336 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:08.275345 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:08.277258 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:08.279023 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:08.279028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:08.279030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:08.279233 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:08.279404 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:08.279405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:08.279405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:08.279494 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:08.279816 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:08.279822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:08.280001 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.280116 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.280303 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260926 06:25:08.280934 3806 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38215
I20260926 06:25:08.281011 3806 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:08.281190 3806 heartbeater.cc:511] Master 127.2.39.254:38215 requested a full tablet report, sending...
I20260926 06:25:08.281540 3323 ts_manager.cc:194] Registered new tserver with Master: 3ce88f7f49cc4d98bd069f23c4449013 (127.2.39.195:36545)
I20260926 06:25:08.281996 3323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:57299
I20260926 06:25:08.286936 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.282027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:44629 (local address 127.2.39.254:38215)
0926 06:25:08.282065 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.282067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.282176 (+ 109us) server_negotiation.cc:407] Connection header received
0926 06:25:08.282244 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.282245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.282252 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.282323 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:08.282613 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.283138 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.283799 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.283980 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.284026 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.284752 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:08.284753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:08.284753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:08.284759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:08.286117 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.286395 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.286396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.286396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.286439 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.286582 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.286582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.286582 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.286679 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:08.286722 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.286828 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.286879 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:25:08.286978 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.281942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38215 (local address 127.2.39.195:44629)
0926 06:25:08.282118 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.282129 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.282191 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.282329 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.282329 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.282492 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:08.282542 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.282543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.283226 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.283227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.283730 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.283732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.284040 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.284041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.284094 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.284433 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:08.284437 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:08.284681 (+ 244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:08.286192 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:08.286193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:08.286193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:08.286334 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:08.286493 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:08.286493 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:08.286493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:08.286546 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:08.286730 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:08.286731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:08.286763 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.286812 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.286911 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260926 06:25:08.291676 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:08.327636 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.316343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.2.39.254:38215)
0926 06:25:08.316423 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.316426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.316439 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:25:08.316478 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.316479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.316492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.316575 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:08.317554 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.318343 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.319593 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.319774 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.319830 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.325239 (+ 5409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:08.325241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:08.325241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:08.325248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:08.326655 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.326971 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.326972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.326972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.327019 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.327181 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.327182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.327182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.327278 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:08.327342 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.327509 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.327568 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 06:25:08.329394 3323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:08.331060 3323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:08.337409 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.333332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:46759 (local address 127.0.0.1:35470)
0926 06:25:08.333479 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.333519 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.333641 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.333826 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.333829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.333831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:08.333893 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.333895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.335150 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.335151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.336096 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.336098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.337108 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.337109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.337197 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.337211 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.337285 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.337334 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":92,"threads_started":1}
I20260926 06:25:08.337450 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.333457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.2.39.194:46759)
0926 06:25:08.333651 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.333656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.333667 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:25:08.333730 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.333732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.333747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.333821 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:08.333980 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.335051 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.336189 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.337023 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.337098 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.337156 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.337309 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.337349 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260926 06:25:08.338021 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.333335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:36545 (local address 127.0.0.1:47604)
0926 06:25:08.333645 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.333656 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.333708 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.334218 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.334219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.334222 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:08.334271 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.334272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.335560 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.335561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.336809 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.336810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.337825 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.337826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.337889 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.337893 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.337921 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.337958 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":75,"thread_start_us":115,"threads_started":1}
I20260926 06:25:08.338209 3597 tablet_service.cc:1467] Processing CreateTablet for tablet 65b55ffde45f40a5bf0e5d5dd8fa9051 (DEFAULT_TABLE table=test-table [id=bc6dea31e89b499e8c8f13fb5b109185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:08.338375 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.333878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.2.39.195:36545)
0926 06:25:08.334033 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.334036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.334047 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:25:08.334093 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.334095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.334111 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.334202 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:08.334611 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.335471 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.336891 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.337732 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.338148 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.338217 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.338242 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.338283 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260926 06:25:08.338370 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.333725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.2.39.193:42181)
0926 06:25:08.333886 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.333892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.334239 (+ 347us) server_negotiation.cc:407] Connection header received
0926 06:25:08.334357 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.334360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.334437 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.334512 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:08.334693 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.335595 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.336914 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.337847 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.337912 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.337970 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.338242 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.338282 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260926 06:25:08.338630 3597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b55ffde45f40a5bf0e5d5dd8fa9051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:08.338706 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.334038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:42181 (local address 127.0.0.1:58976)
0926 06:25:08.334167 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.334177 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.334271 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.334569 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.334569 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.334571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:08.334607 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.334608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.335685 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.335686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.336807 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.336810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.338005 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.338006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.338146 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.338151 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.338589 (+ 438us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.338634 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260926 06:25:08.339075 3737 tablet_service.cc:1467] Processing CreateTablet for tablet 65b55ffde45f40a5bf0e5d5dd8fa9051 (DEFAULT_TABLE table=test-table [id=bc6dea31e89b499e8c8f13fb5b109185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:08.339547 3737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b55ffde45f40a5bf0e5d5dd8fa9051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:08.341356 3829 tablet_bootstrap.cc:492] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: Bootstrap starting.
I20260926 06:25:08.342051 3829 tablet_bootstrap.cc:654] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:08.342272 3460 tablet_service.cc:1467] Processing CreateTablet for tablet 65b55ffde45f40a5bf0e5d5dd8fa9051 (DEFAULT_TABLE table=test-table [id=bc6dea31e89b499e8c8f13fb5b109185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:08.342376 3829 log.cc:824] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:08.342615 3460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65b55ffde45f40a5bf0e5d5dd8fa9051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:08.343192 3829 tablet_bootstrap.cc:492] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: No bootstrap required, opened a new log
I20260926 06:25:08.343264 3829 ts_tablet_manager.cc:1397] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:08.345301 3829 raft_consensus.cc:348] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } }
I20260926 06:25:08.345504 3829 raft_consensus.cc:374] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:08.345543 3829 raft_consensus.cc:729] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e03a97b15945709c80d501ff1b3135, State: Initialized, Role: FOLLOWER
I20260926 06:25:08.345749 3829 consensus_queue.cc:260] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } }
I20260926 06:25:08.345757 3831 tablet_bootstrap.cc:492] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: Bootstrap starting.
I20260926 06:25:08.346062 3666 heartbeater.cc:503] Master 127.2.39.254:38215 was elected leader, sending a full tablet report...
I20260926 06:25:08.346046 3829 ts_tablet_manager.cc:1428] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:25:08.346465 3833 tablet_bootstrap.cc:492] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: Bootstrap starting.
I20260926 06:25:08.346946 3831 tablet_bootstrap.cc:654] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:08.346987 3833 tablet_bootstrap.cc:654] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:08.347280 3831 log.cc:824] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:08.347316 3833 log.cc:824] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:08.347961 3833 tablet_bootstrap.cc:492] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: No bootstrap required, opened a new log
I20260926 06:25:08.347995 3831 tablet_bootstrap.cc:492] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: No bootstrap required, opened a new log
I20260926 06:25:08.348034 3833 ts_tablet_manager.cc:1397] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:08.348060 3831 ts_tablet_manager.cc:1397] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:25:08.349756 3833 raft_consensus.cc:348] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } }
I20260926 06:25:08.349910 3833 raft_consensus.cc:374] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:08.349934 3833 raft_consensus.cc:729] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8988947728c4d9a96b7ea6fdd60ba07, State: Initialized, Role: FOLLOWER
I20260926 06:25:08.349958 3831 raft_consensus.cc:348] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } }
I20260926 06:25:08.350049 3833 consensus_queue.cc:260] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } }
I20260926 06:25:08.350122 3831 raft_consensus.cc:374] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:08.350163 3831 raft_consensus.cc:729] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce88f7f49cc4d98bd069f23c4449013, State: Initialized, Role: FOLLOWER
I20260926 06:25:08.350303 3833 ts_tablet_manager.cc:1428] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:25:08.350275 3831 consensus_queue.cc:260] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } }
I20260926 06:25:08.350317 3526 heartbeater.cc:503] Master 127.2.39.254:38215 was elected leader, sending a full tablet report...
I20260926 06:25:08.350580 3806 heartbeater.cc:503] Master 127.2.39.254:38215 was elected leader, sending a full tablet report...
I20260926 06:25:08.350630 3831 ts_tablet_manager.cc:1428] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260926 06:25:08.359782 3667 tablet.cc:2369] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:25:08.388164 3527 tablet.cc:2369] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:25:08.527902 3807 tablet.cc:2369] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:25:08.551756 3836 raft_consensus.cc:482] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:08.551927 3836 raft_consensus.cc:504] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } }
I20260926 06:25:08.552304 3836 leader_election.cc:288] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ce88f7f49cc4d98bd069f23c4449013 (127.2.39.195:36545), 25e03a97b15945709c80d501ff1b3135 (127.2.39.194:46759)
I20260926 06:25:08.555655 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.552660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:53739 (local address 127.2.39.195:36545)
0926 06:25:08.552732 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.552735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.552914 (+ 179us) server_negotiation.cc:407] Connection header received
0926 06:25:08.553021 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.553022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.553050 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.553113 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:08.553268 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.553803 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.554861 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.555263 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.555349 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.555358 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.555527 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.555565 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:25:08.555668 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.552672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:46759 (local address 127.2.39.193:37887)
0926 06:25:08.552828 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.552841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.552906 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.553075 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.553076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.553079 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:08.553164 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.553167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.553884 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.553885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.554733 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.554735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.555384 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.555385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.555459 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.555464 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.555535 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.555579 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260926 06:25:08.555655 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.552760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:37887 (local address 127.2.39.194:46759)
0926 06:25:08.552800 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.552802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.552934 (+ 132us) server_negotiation.cc:407] Connection header received
0926 06:25:08.552983 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.552985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.552997 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.553052 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:08.553242 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.553806 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.554900 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.555276 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.555384 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.555393 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.555543 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.555584 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:25:08.555907 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.552657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:36545 (local address 127.2.39.193:53739)
0926 06:25:08.552816 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.552827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.552930 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.553128 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.553128 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.553135 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:08.553182 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.553184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.553914 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.553915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.554728 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.554730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.555358 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.555359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.555426 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.555469 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.555520 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.555560 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":42,"thread_start_us":74,"threads_started":1}
I20260926 06:25:08.556072 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b55ffde45f40a5bf0e5d5dd8fa9051" candidate_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e03a97b15945709c80d501ff1b3135" is_pre_election: true
I20260926 06:25:08.556246 3620 raft_consensus.cc:2454] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8988947728c4d9a96b7ea6fdd60ba07 in term 0.
I20260926 06:25:08.556401 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b55ffde45f40a5bf0e5d5dd8fa9051" candidate_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce88f7f49cc4d98bd069f23c4449013" is_pre_election: true
I20260926 06:25:08.556488 3411 leader_election.cc:302] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e03a97b15945709c80d501ff1b3135, a8988947728c4d9a96b7ea6fdd60ba07; no voters:
I20260926 06:25:08.556550 3760 raft_consensus.cc:2454] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8988947728c4d9a96b7ea6fdd60ba07 in term 0.
I20260926 06:25:08.556648 3836 raft_consensus.cc:2781] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:08.556738 3836 raft_consensus.cc:482] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:08.556775 3836 raft_consensus.cc:3037] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:08.557706 3836 raft_consensus.cc:504] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } }
I20260926 06:25:08.557840 3836 leader_election.cc:288] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [CANDIDATE]: Term 1 election: Requested vote from peers 3ce88f7f49cc4d98bd069f23c4449013 (127.2.39.195:36545), 25e03a97b15945709c80d501ff1b3135 (127.2.39.194:46759)
I20260926 06:25:08.557991 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b55ffde45f40a5bf0e5d5dd8fa9051" candidate_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e03a97b15945709c80d501ff1b3135"
I20260926 06:25:08.558069 3620 raft_consensus.cc:3037] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:08.558050 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65b55ffde45f40a5bf0e5d5dd8fa9051" candidate_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce88f7f49cc4d98bd069f23c4449013"
I20260926 06:25:08.558143 3760 raft_consensus.cc:3037] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:08.558898 3620 raft_consensus.cc:2454] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8988947728c4d9a96b7ea6fdd60ba07 in term 1.
I20260926 06:25:08.559082 3411 leader_election.cc:302] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e03a97b15945709c80d501ff1b3135, a8988947728c4d9a96b7ea6fdd60ba07; no voters:
I20260926 06:25:08.559128 3760 raft_consensus.cc:2454] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8988947728c4d9a96b7ea6fdd60ba07 in term 1.
I20260926 06:25:08.559192 3836 raft_consensus.cc:2781] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:08.559321 3836 raft_consensus.cc:686] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 LEADER]: Becoming Leader. State: Replica: a8988947728c4d9a96b7ea6fdd60ba07, State: Running, Role: LEADER
I20260926 06:25:08.559417 3836 consensus_queue.cc:237] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } }
I20260926 06:25:08.560189 3323 catalog_manager.cc:5495] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8988947728c4d9a96b7ea6fdd60ba07 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:08.621150 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.618225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.2.39.193:42181)
0926 06:25:08.618278 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.618280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.618290 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:08.618426 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.618427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.618467 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.618545 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:08.618797 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.619272 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.620298 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.620516 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.620601 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.620809 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:08.620917 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:08.621003 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.621044 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.621065 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:25:08.622947 3760 raft_consensus.cc:1261] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 FOLLOWER]: Refusing update from remote peer a8988947728c4d9a96b7ea6fdd60ba07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:08.622947 3620 raft_consensus.cc:1261] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135 [term 1 FOLLOWER]: Refusing update from remote peer a8988947728c4d9a96b7ea6fdd60ba07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:08.623353 3836 consensus_queue.cc:1035] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:08.623510 3840 consensus_queue.cc:1035] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:08.625782 3843 mvcc.cc:205] Tried to move back new op lower bound from 7333494409716912128 to 7333494409460060160. Current Snapshot: MvccSnapshot[applied={T|T < 7333494409716912128}]
I20260926 06:25:08.626402 3845 mvcc.cc:205] Tried to move back new op lower bound from 7333494409716912128 to 7333494409460060160. Current Snapshot: MvccSnapshot[applied={T|T < 7333494409716912128}]
I20260926 06:25:08.626520 3844 mvcc.cc:205] Tried to move back new op lower bound from 7333494409716912128 to 7333494409460060160. Current Snapshot: MvccSnapshot[applied={T|T < 7333494409716912128}]
I20260926 06:25:08.629609 3323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:08.629714 3323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:08.630525 3323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb: Sending DeleteTablet for 3 replicas of tablet 65b55ffde45f40a5bf0e5d5dd8fa9051
I20260926 06:25:08.630894 3460 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b55ffde45f40a5bf0e5d5dd8fa9051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:08 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:58976
I20260926 06:25:08.630951 3597 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b55ffde45f40a5bf0e5d5dd8fa9051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:08 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:35470
I20260926 06:25:08.631163 3737 tablet_service.cc:1514] Processing DeleteTablet for tablet 65b55ffde45f40a5bf0e5d5dd8fa9051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:08 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:47604
I20260926 06:25:08.631156 3323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:08.631299 3323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:08.631716 3855 tablet_replica.cc:330] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: stopping tablet replica
I20260926 06:25:08.631841 3855 raft_consensus.cc:2229] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:08.631953 3855 raft_consensus.cc:2258] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:08.632436 3855 ts_tablet_manager.cc:1905] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:08.632756 3597 tablet_service.cc:1467] Processing CreateTablet for tablet 2221fecadf814b55bd0eda3de6b7cd30 (DEFAULT_TABLE table=test-table [id=123a67eb31334a6a8cf09b6e527b7b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:08.632787 3737 tablet_service.cc:1467] Processing CreateTablet for tablet 2221fecadf814b55bd0eda3de6b7cd30 (DEFAULT_TABLE table=test-table [id=123a67eb31334a6a8cf09b6e527b7b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:08.632890 3597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2221fecadf814b55bd0eda3de6b7cd30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:08.632892 3737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2221fecadf814b55bd0eda3de6b7cd30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:08.634212 3854 tablet_replica.cc:330] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: stopping tablet replica
I20260926 06:25:08.634244 3829 tablet_bootstrap.cc:492] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135: Bootstrap starting.
I20260926 06:25:08.634308 3854 raft_consensus.cc:2229] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:08.634302 3831 tablet_bootstrap.cc:492] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013: Bootstrap starting.
I20260926 06:25:08.634402 3854 raft_consensus.cc:2258] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:08.634634 3853 tablet_replica.cc:330] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: stopping tablet replica
I20260926 06:25:08.634668 3829 tablet_bootstrap.cc:654] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:08.634737 3853 raft_consensus.cc:2229] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:08.634734 3831 tablet_bootstrap.cc:654] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:08.634887 3853 raft_consensus.cc:2258] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:08.635281 3460 tablet_service.cc:1467] Processing CreateTablet for tablet 2221fecadf814b55bd0eda3de6b7cd30 (DEFAULT_TABLE table=test-table [id=123a67eb31334a6a8cf09b6e527b7b70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:08.635370 3853 ts_tablet_manager.cc:1905] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:08.635407 3460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2221fecadf814b55bd0eda3de6b7cd30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:08.636840 3833 tablet_bootstrap.cc:492] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07: Bootstrap starting.
I20260926 06:25:08.636853 3853 ts_tablet_manager.cc:1918] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:08.636957 3853 log.cc:1196] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/65b55ffde45f40a5bf0e5d5dd8fa9051
I20260926 06:25:08.637274 3833 tablet_bootstrap.cc:654] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:08.637473 3853 ts_tablet_manager.cc:1939] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P a8988947728c4d9a96b7ea6fdd60ba07: Deleting consensus metadata
I20260926 06:25:08.638439 3854 ts_tablet_manager.cc:1905] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:08.638370 3855 ts_tablet_manager.cc:1918] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:08.638655 3855 log.cc:1196] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/65b55ffde45f40a5bf0e5d5dd8fa9051
I20260926 06:25:08.639063 3855 ts_tablet_manager.cc:1939] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 3ce88f7f49cc4d98bd069f23c4449013: Deleting consensus metadata
I20260926 06:25:08.639531 3309 catalog_manager.cc:4841] TS 3ce88f7f49cc4d98bd069f23c4449013 (127.2.39.195:36545): tablet 65b55ffde45f40a5bf0e5d5dd8fa9051 (table test-table [id=bc6dea31e89b499e8c8f13fb5b109185]) successfully deleted
I20260926 06:25:08.639722 3831 tablet_bootstrap.cc:492] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013: No bootstrap required, opened a new log
I20260926 06:25:08.639777 3831 ts_tablet_manager.cc:1397] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 06:25:08.639954 3831 raft_consensus.cc:348] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.640055 3831 raft_consensus.cc:374] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:08.640086 3831 raft_consensus.cc:729] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce88f7f49cc4d98bd069f23c4449013, State: Initialized, Role: FOLLOWER
I20260926 06:25:08.640125 3854 ts_tablet_manager.cc:1918] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:08.640146 3831 consensus_queue.cc:260] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.640192 3854 log.cc:1196] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/65b55ffde45f40a5bf0e5d5dd8fa9051
I20260926 06:25:08.640264 3831 ts_tablet_manager.cc:1428] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:08.640818 3854 ts_tablet_manager.cc:1939] T 65b55ffde45f40a5bf0e5d5dd8fa9051 P 25e03a97b15945709c80d501ff1b3135: Deleting consensus metadata
I20260926 06:25:08.641253 3833 tablet_bootstrap.cc:492] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07: No bootstrap required, opened a new log
I20260926 06:25:08.641306 3833 ts_tablet_manager.cc:1397] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 06:25:08.641325 3309 catalog_manager.cc:4841] TS a8988947728c4d9a96b7ea6fdd60ba07 (127.2.39.193:42181): tablet 65b55ffde45f40a5bf0e5d5dd8fa9051 (table test-table [id=bc6dea31e89b499e8c8f13fb5b109185]) successfully deleted
I20260926 06:25:08.641294 3307 catalog_manager.cc:4841] TS 25e03a97b15945709c80d501ff1b3135 (127.2.39.194:46759): tablet 65b55ffde45f40a5bf0e5d5dd8fa9051 (table test-table [id=bc6dea31e89b499e8c8f13fb5b109185]) successfully deleted
I20260926 06:25:08.641456 3833 raft_consensus.cc:348] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.641536 3833 raft_consensus.cc:374] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:08.641561 3833 raft_consensus.cc:729] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8988947728c4d9a96b7ea6fdd60ba07, State: Initialized, Role: FOLLOWER
I20260926 06:25:08.641639 3833 consensus_queue.cc:260] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.641726 3833 ts_tablet_manager.cc:1428] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:08.648972 3829 tablet_bootstrap.cc:492] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135: No bootstrap required, opened a new log
I20260926 06:25:08.649041 3829 ts_tablet_manager.cc:1397] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20260926 06:25:08.649235 3829 raft_consensus.cc:348] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.649318 3829 raft_consensus.cc:374] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:08.649356 3829 raft_consensus.cc:729] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e03a97b15945709c80d501ff1b3135, State: Initialized, Role: FOLLOWER
I20260926 06:25:08.649399 3829 consensus_queue.cc:260] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.649515 3829 ts_tablet_manager.cc:1428] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:08.651916 3836 raft_consensus.cc:482] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:08.652004 3836 raft_consensus.cc:504] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.652137 3836 leader_election.cc:288] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25e03a97b15945709c80d501ff1b3135 (127.2.39.194:46759), 3ce88f7f49cc4d98bd069f23c4449013 (127.2.39.195:36545)
I20260926 06:25:08.652262 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221fecadf814b55bd0eda3de6b7cd30" candidate_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e03a97b15945709c80d501ff1b3135" is_pre_election: true
I20260926 06:25:08.652331 3620 raft_consensus.cc:2454] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8988947728c4d9a96b7ea6fdd60ba07 in term 0.
I20260926 06:25:08.652307 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221fecadf814b55bd0eda3de6b7cd30" candidate_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce88f7f49cc4d98bd069f23c4449013" is_pre_election: true
I20260926 06:25:08.652400 3760 raft_consensus.cc:2454] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8988947728c4d9a96b7ea6fdd60ba07 in term 0.
I20260926 06:25:08.652526 3411 leader_election.cc:302] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e03a97b15945709c80d501ff1b3135, a8988947728c4d9a96b7ea6fdd60ba07; no voters:
I20260926 06:25:08.652659 3836 raft_consensus.cc:2781] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:08.652719 3836 raft_consensus.cc:482] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:08.652751 3836 raft_consensus.cc:3037] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:08.653648 3836 raft_consensus.cc:504] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.653789 3836 leader_election.cc:288] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [CANDIDATE]: Term 1 election: Requested vote from peers 25e03a97b15945709c80d501ff1b3135 (127.2.39.194:46759), 3ce88f7f49cc4d98bd069f23c4449013 (127.2.39.195:36545)
I20260926 06:25:08.653909 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221fecadf814b55bd0eda3de6b7cd30" candidate_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e03a97b15945709c80d501ff1b3135"
I20260926 06:25:08.653927 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2221fecadf814b55bd0eda3de6b7cd30" candidate_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ce88f7f49cc4d98bd069f23c4449013"
I20260926 06:25:08.653975 3620 raft_consensus.cc:3037] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:08.653986 3760 raft_consensus.cc:3037] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:08.654747 3760 raft_consensus.cc:2454] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8988947728c4d9a96b7ea6fdd60ba07 in term 1.
I20260926 06:25:08.654773 3620 raft_consensus.cc:2454] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8988947728c4d9a96b7ea6fdd60ba07 in term 1.
I20260926 06:25:08.654979 3413 leader_election.cc:302] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ce88f7f49cc4d98bd069f23c4449013, a8988947728c4d9a96b7ea6fdd60ba07; no voters:
I20260926 06:25:08.655102 3836 raft_consensus.cc:2781] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:08.655172 3836 raft_consensus.cc:686] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 LEADER]: Becoming Leader. State: Replica: a8988947728c4d9a96b7ea6fdd60ba07, State: Running, Role: LEADER
I20260926 06:25:08.655226 3836 consensus_queue.cc:237] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.655750 3322 catalog_manager.cc:5495] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8988947728c4d9a96b7ea6fdd60ba07 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } health_report { overall_health: UNKNOWN } } }
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403907, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:08.693909 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.686363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51312 (local address 127.2.39.254:38215)
0926 06:25:08.686438 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.686440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.686592 (+ 152us) server_negotiation.cc:407] Connection header received
0926 06:25:08.686674 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.686674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.686686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.686762 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:08.687274 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.687757 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.688446 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.688630 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.688714 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.691077 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:08.691079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:08.691079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:08.691087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:08.692582 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.693038 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.693039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.693040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.693099 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.693316 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.693317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.693317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.693435 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:08.693515 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.693682 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.693770 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:25:08.693985 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.686313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38215 (local address 127.0.0.1:51312)
0926 06:25:08.686484 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.686518 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.686612 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.686770 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.686771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.687136 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:08.687184 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.687187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.687848 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.687849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.688365 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.688366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.688713 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.688714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.688763 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.689340 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:08.689345 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:08.690963 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:08.692691 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:08.692698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:08.692700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:08.692943 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:08.693182 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:08.693182 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:08.693183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:08.693251 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:08.693506 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:08.693512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:08.693557 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.693660 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.693848 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":80,"thread_start_us":94,"threads_started":1}
I20260926 06:25:08.696501 3322 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:51312:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 06:25:08.696725 3322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:08.698261 3460 tablet_service.cc:1467] Processing CreateTablet for tablet ba691b1bff4040dcaf44d75b1a5ade6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4bf280b119024067ba026453193250b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:25:08.698359 3597 tablet_service.cc:1467] Processing CreateTablet for tablet ba691b1bff4040dcaf44d75b1a5ade6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4bf280b119024067ba026453193250b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:25:08.698372 3737 tablet_service.cc:1467] Processing CreateTablet for tablet ba691b1bff4040dcaf44d75b1a5ade6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4bf280b119024067ba026453193250b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:25:08.698436 3460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba691b1bff4040dcaf44d75b1a5ade6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:08.698493 3737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba691b1bff4040dcaf44d75b1a5ade6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:08.698508 3597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba691b1bff4040dcaf44d75b1a5ade6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:08.700006 3833 tablet_bootstrap.cc:492] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: Bootstrap starting.
I20260926 06:25:08.700006 3831 tablet_bootstrap.cc:492] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: Bootstrap starting.
I20260926 06:25:08.700302 3833 tablet_bootstrap.cc:654] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:08.700415 3831 tablet_bootstrap.cc:654] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:08.700728 3829 tablet_bootstrap.cc:492] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: Bootstrap starting.
I20260926 06:25:08.701126 3829 tablet_bootstrap.cc:654] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:08.701156 3833 tablet_bootstrap.cc:492] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: No bootstrap required, opened a new log
I20260926 06:25:08.701200 3833 ts_tablet_manager.cc:1397] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:08.701346 3833 raft_consensus.cc:348] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.701422 3833 raft_consensus.cc:374] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:08.701443 3833 raft_consensus.cc:729] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8988947728c4d9a96b7ea6fdd60ba07, State: Initialized, Role: FOLLOWER
I20260926 06:25:08.701493 3833 consensus_queue.cc:260] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.701604 3833 ts_tablet_manager.cc:1428] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:08.701711 3836 tablet_replica.cc:439] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } }
I20260926 06:25:08.701777 3836 tablet_replica.cc:442] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:08.701861 3829 tablet_bootstrap.cc:492] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: No bootstrap required, opened a new log
I20260926 06:25:08.701900 3829 ts_tablet_manager.cc:1397] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:08.702028 3829 raft_consensus.cc:348] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.702098 3829 raft_consensus.cc:374] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:08.702124 3829 raft_consensus.cc:729] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e03a97b15945709c80d501ff1b3135, State: Initialized, Role: FOLLOWER
I20260926 06:25:08.702175 3829 consensus_queue.cc:260] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.702283 3831 tablet_bootstrap.cc:492] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: No bootstrap required, opened a new log
I20260926 06:25:08.702284 3829 ts_tablet_manager.cc:1428] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:08.702322 3831 ts_tablet_manager.cc:1397] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:08.702486 3831 raft_consensus.cc:348] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.702560 3831 raft_consensus.cc:374] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:08.702577 3831 raft_consensus.cc:729] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ce88f7f49cc4d98bd069f23c4449013, State: Initialized, Role: FOLLOWER
I20260926 06:25:08.702610 3831 consensus_queue.cc:260] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.702684 3831 ts_tablet_manager.cc:1428] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:08.702811 3832 tablet_replica.cc:439] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } }
I20260926 06:25:08.702912 3832 tablet_replica.cc:442] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:08.703060 3837 tablet_replica.cc:439] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } }
I20260926 06:25:08.703169 3837 tablet_replica.cc:442] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:08.721870 3837 raft_consensus.cc:482] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:08.721967 3837 raft_consensus.cc:504] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.722297 3837 leader_election.cc:288] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8988947728c4d9a96b7ea6fdd60ba07 (127.2.39.193:42181), 25e03a97b15945709c80d501ff1b3135 (127.2.39.194:46759)
I20260926 06:25:08.726083 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.722476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:46759 (local address 127.2.39.195:42673)
0926 06:25:08.722530 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.722541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.722584 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.722806 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.722807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.722809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:08.722877 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.722879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.723743 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.723744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.725042 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.725045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.725811 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.725812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.725893 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.725897 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.725955 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.725997 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":42}
I20260926 06:25:08.726084 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.722519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:42673 (local address 127.2.39.194:46759)
0926 06:25:08.722595 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.722597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.722605 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:08.722633 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.722634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.722643 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.722702 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:08.722974 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.723671 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.725141 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.725728 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.725808 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.725818 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.725973 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.726015 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 06:25:08.726320 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba691b1bff4040dcaf44d75b1a5ade6e" candidate_uuid: "3ce88f7f49cc4d98bd069f23c4449013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e03a97b15945709c80d501ff1b3135" is_pre_election: true
I20260926 06:25:08.726404 3620 raft_consensus.cc:2454] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ce88f7f49cc4d98bd069f23c4449013 in term 0.
I20260926 06:25:08.726578 3691 leader_election.cc:302] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e03a97b15945709c80d501ff1b3135, 3ce88f7f49cc4d98bd069f23c4449013; no voters:
I20260926 06:25:08.726716 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.722500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:42181 (local address 127.2.39.195:41745)
0926 06:25:08.722667 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.722675 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.722724 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.722961 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.722961 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.722963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:08.723033 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.723035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.723678 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.723679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.725479 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.725481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.726461 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.726462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.726539 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.726543 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.726596 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.726634 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":26,"thread_start_us":108,"threads_started":1}
I20260926 06:25:08.726713 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.722755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:41745 (local address 127.2.39.193:42181)
0926 06:25:08.722820 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.722822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.722830 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:08.722879 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.722880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.722887 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.722956 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:08.723114 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.723590 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.725562 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.726116 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.726195 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.726205 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.726613 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.726652 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 06:25:08.726835 3837 raft_consensus.cc:2781] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:08.726874 3837 raft_consensus.cc:482] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:08.726920 3837 raft_consensus.cc:3037] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:08.727779 3837 raft_consensus.cc:504] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.727928 3837 leader_election.cc:288] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [CANDIDATE]: Term 1 election: Requested vote from peers a8988947728c4d9a96b7ea6fdd60ba07 (127.2.39.193:42181), 25e03a97b15945709c80d501ff1b3135 (127.2.39.194:46759)
I20260926 06:25:08.728116 3479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba691b1bff4040dcaf44d75b1a5ade6e" candidate_uuid: "3ce88f7f49cc4d98bd069f23c4449013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8988947728c4d9a96b7ea6fdd60ba07"
I20260926 06:25:08.728094 3480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba691b1bff4040dcaf44d75b1a5ade6e" candidate_uuid: "3ce88f7f49cc4d98bd069f23c4449013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" is_pre_election: true
I20260926 06:25:08.728183 3620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba691b1bff4040dcaf44d75b1a5ade6e" candidate_uuid: "3ce88f7f49cc4d98bd069f23c4449013" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25e03a97b15945709c80d501ff1b3135"
I20260926 06:25:08.728219 3480 raft_consensus.cc:2454] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ce88f7f49cc4d98bd069f23c4449013 in term 0.
I20260926 06:25:08.728247 3620 raft_consensus.cc:3037] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:08.729061 3620 raft_consensus.cc:2454] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ce88f7f49cc4d98bd069f23c4449013 in term 1.
I20260926 06:25:08.729235 3691 leader_election.cc:302] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25e03a97b15945709c80d501ff1b3135, 3ce88f7f49cc4d98bd069f23c4449013; no voters:
I20260926 06:25:08.729374 3837 raft_consensus.cc:2781] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:08.729527 3837 raft_consensus.cc:686] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 LEADER]: Becoming Leader. State: Replica: 3ce88f7f49cc4d98bd069f23c4449013, State: Running, Role: LEADER
I20260926 06:25:08.729722 3837 consensus_queue.cc:237] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } }
I20260926 06:25:08.730509 3870 tablet_replica.cc:439] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: TxnStatusTablet state changed. Reason: New leader 3ce88f7f49cc4d98bd069f23c4449013. Latest consensus state: current_term: 1 leader_uuid: "3ce88f7f49cc4d98bd069f23c4449013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } }
I20260926 06:25:08.730612 3870 tablet_replica.cc:442] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:25:08.730926 3872 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 06:25:08.731029 3872 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 06:25:08.731189 3322 catalog_manager.cc:5495] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ce88f7f49cc4d98bd069f23c4449013 (127.2.39.195). New cstate: current_term: 1 leader_uuid: "3ce88f7f49cc4d98bd069f23c4449013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } health_report { overall_health: HEALTHY } } }
I20260926 06:25:08.741328 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.738120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.2.39.195:36545)
0926 06:25:08.738159 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.738162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.738171 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:25:08.738256 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.738257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.738274 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.738340 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:08.738473 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.739186 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.740370 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.740639 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.740717 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.740944 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:08.741061 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:08.741151 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.741233 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.741255 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:25:08.741364 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.738049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:36545 (local address 127.0.0.1:34690)
0926 06:25:08.738130 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.738141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.738184 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.738352 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.738353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.738354 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:08.738405 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.738407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.739295 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.739296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.740272 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.740275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.740736 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.740737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.740812 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.740827 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:08.741157 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:08.741165 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.741230 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.741283 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 06:25:08.742570 3620 raft_consensus.cc:1261] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135 [term 1 FOLLOWER]: Refusing update from remote peer 3ce88f7f49cc4d98bd069f23c4449013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:08.742594 3479 raft_consensus.cc:3037] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:08.742801 3837 consensus_queue.cc:1035] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:08.743388 3832 tablet_replica.cc:439] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: TxnStatusTablet state changed. Reason: New leader 3ce88f7f49cc4d98bd069f23c4449013. Latest consensus state: current_term: 1 leader_uuid: "3ce88f7f49cc4d98bd069f23c4449013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } }
I20260926 06:25:08.743482 3832 tablet_replica.cc:442] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:08.743559 3479 raft_consensus.cc:1261] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 FOLLOWER]: Refusing update from remote peer 3ce88f7f49cc4d98bd069f23c4449013: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:08.743819 3837 consensus_queue.cc:1035] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:08.744427 3836 tablet_replica.cc:439] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: TxnStatusTablet state changed. Reason: New leader 3ce88f7f49cc4d98bd069f23c4449013. Latest consensus state: current_term: 1 leader_uuid: "3ce88f7f49cc4d98bd069f23c4449013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } }
I20260926 06:25:08.744516 3836 tablet_replica.cc:442] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:08.744616 3837 tablet_replica.cc:439] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ce88f7f49cc4d98bd069f23c4449013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } }
I20260926 06:25:08.744699 3837 tablet_replica.cc:442] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:25:08.745124 3832 tablet_replica.cc:439] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ce88f7f49cc4d98bd069f23c4449013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } }
I20260926 06:25:08.745203 3832 tablet_replica.cc:442] T ba691b1bff4040dcaf44d75b1a5ade6e P 25e03a97b15945709c80d501ff1b3135: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:08.746927 3836 tablet_replica.cc:439] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ce88f7f49cc4d98bd069f23c4449013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } }
I20260926 06:25:08.747008 3836 tablet_replica.cc:442] T ba691b1bff4040dcaf44d75b1a5ade6e P a8988947728c4d9a96b7ea6fdd60ba07: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:08.747108 3837 tablet_replica.cc:439] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ce88f7f49cc4d98bd069f23c4449013" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8988947728c4d9a96b7ea6fdd60ba07" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42181 } } peers { permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 } } peers { permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 } } }
I20260926 06:25:08.747189 3837 tablet_replica.cc:442] T ba691b1bff4040dcaf44d75b1a5ade6e P 3ce88f7f49cc4d98bd069f23c4449013: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:25:08.752492 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.748916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:34935 (local address 127.2.39.195:36545)
0926 06:25:08.749054 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.749056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.749124 (+ 68us) server_negotiation.cc:407] Connection header received
0926 06:25:08.749190 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.749190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.749200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.749254 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:08.749382 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.750147 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.751302 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.751989 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.752058 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.752248 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:08.752280 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:08.752332 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.752402 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.752421 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260926 06:25:08.752553 3880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.748912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:36545 (local address 127.2.39.193:34935)
0926 06:25:08.749065 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.749077 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.749140 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.749254 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.749255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.749257 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:08.749319 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.749321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.750223 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.750224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.751212 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.751214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.752073 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.752074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.752146 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.752161 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:08.752347 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:08.752349 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.752393 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.752441 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260926 06:25:08.754704 3760 raft_consensus.cc:1261] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 FOLLOWER]: Refusing update from remote peer a8988947728c4d9a96b7ea6fdd60ba07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:08.754920 3620 raft_consensus.cc:1261] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135 [term 1 FOLLOWER]: Refusing update from remote peer a8988947728c4d9a96b7ea6fdd60ba07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:08.754914 3848 consensus_queue.cc:1035] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ce88f7f49cc4d98bd069f23c4449013" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:08.755146 3848 consensus_queue.cc:1035] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25e03a97b15945709c80d501ff1b3135" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:08.765010 3812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.761140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:42181 (local address 127.2.39.195:50337)
0926 06:25:08.761248 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:08.761260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:08.761305 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:08.761530 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:08.761531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:08.761532 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:08.761611 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.761613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.762634 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.762635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.763577 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:08.763579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.764587 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.764588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:08.764654 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.764668 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:08.764844 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:08.764845 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:25:08.764882 (+ 37us) client_negotiation.cc:240] Negotiation successful
0926 06:25:08.764932 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 06:25:08.765254 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.761229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:50337 (local address 127.2.39.193:42181)
0926 06:25:08.761265 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.761267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.761368 (+ 101us) server_negotiation.cc:407] Connection header received
0926 06:25:08.761390 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.761390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.761402 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.761446 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:08.761684 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.762414 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.764179 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.764489 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.764553 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.764749 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:08.764783 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:08.764824 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.765171 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.765190 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:25:08.772845 3320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:08.772946 3320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:08.773614 3320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4510fb0b0e740f0855362bd27eb24bb: Sending DeleteTablet for 3 replicas of tablet 2221fecadf814b55bd0eda3de6b7cd30
I20260926 06:25:08.773922 3460 tablet_service.cc:1514] Processing DeleteTablet for tablet 2221fecadf814b55bd0eda3de6b7cd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:08 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:58976
I20260926 06:25:08.774044 3597 tablet_service.cc:1514] Processing DeleteTablet for tablet 2221fecadf814b55bd0eda3de6b7cd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:08 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:35470
I20260926 06:25:08.774155 3854 tablet_replica.cc:330] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135: stopping tablet replica
I20260926 06:25:08.774201 3854 raft_consensus.cc:2229] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:08.774256 3854 raft_consensus.cc:2258] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:08.774540 3853 tablet_replica.cc:330] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07: stopping tablet replica
I20260926 06:25:08.774602 3853 raft_consensus.cc:2229] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:08.774648 3853 raft_consensus.cc:2258] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:08.774847 3737 tablet_service.cc:1514] Processing DeleteTablet for tablet 2221fecadf814b55bd0eda3de6b7cd30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:08 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:47604
I20260926 06:25:08.775063 3853 ts_tablet_manager.cc:1905] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:08.775049 3855 tablet_replica.cc:330] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013: stopping tablet replica
I20260926 06:25:08.775143 3855 raft_consensus.cc:2229] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:08.775225 3855 raft_consensus.cc:2258] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:08.775689 3854 ts_tablet_manager.cc:1905] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:08.777467 3854 ts_tablet_manager.cc:1918] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:08.777524 3854 log.cc:1196] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/2221fecadf814b55bd0eda3de6b7cd30
I20260926 06:25:08.777853 3855 ts_tablet_manager.cc:1905] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:08.779006 3853 ts_tablet_manager.cc:1918] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:08.779059 3853 log.cc:1196] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/2221fecadf814b55bd0eda3de6b7cd30
I20260926 06:25:08.779279 3853 ts_tablet_manager.cc:1939] T 2221fecadf814b55bd0eda3de6b7cd30 P a8988947728c4d9a96b7ea6fdd60ba07: Deleting consensus metadata
I20260926 06:25:08.779397 3855 ts_tablet_manager.cc:1918] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:08.779445 3855 log.cc:1196] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/2221fecadf814b55bd0eda3de6b7cd30
I20260926 06:25:08.779587 3309 catalog_manager.cc:4841] TS a8988947728c4d9a96b7ea6fdd60ba07 (127.2.39.193:42181): tablet 2221fecadf814b55bd0eda3de6b7cd30 (table test-table [id=123a67eb31334a6a8cf09b6e527b7b70]) successfully deleted
I20260926 06:25:08.779649 3855 ts_tablet_manager.cc:1939] T 2221fecadf814b55bd0eda3de6b7cd30 P 3ce88f7f49cc4d98bd069f23c4449013: Deleting consensus metadata
I20260926 06:25:08.779853 3854 ts_tablet_manager.cc:1939] T 2221fecadf814b55bd0eda3de6b7cd30 P 25e03a97b15945709c80d501ff1b3135: Deleting consensus metadata
I20260926 06:25:08.779986 3309 catalog_manager.cc:4841] TS 3ce88f7f49cc4d98bd069f23c4449013 (127.2.39.195:36545): tablet 2221fecadf814b55bd0eda3de6b7cd30 (table test-table [id=123a67eb31334a6a8cf09b6e527b7b70]) successfully deleted
I20260926 06:25:08.780169 3307 catalog_manager.cc:4841] TS 25e03a97b15945709c80d501ff1b3135 (127.2.39.194:46759): tablet 2221fecadf814b55bd0eda3de6b7cd30 (table test-table [id=123a67eb31334a6a8cf09b6e527b7b70]) successfully deleted
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403908, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:25:08.789356 3826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.776066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59558 (local address 127.2.39.193:42181)
0926 06:25:08.776103 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.776105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.780203 (+ 4098us) server_negotiation.cc:407] Connection header received
0926 06:25:08.780292 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.780293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.780305 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.780360 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:08.781488 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.782311 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.783043 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.783343 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.783415 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.786458 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:08.786466 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:08.786467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:08.786474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:08.788266 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.788656 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.788658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.788659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.788712 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.788922 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.788922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.788923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.789043 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:08.789139 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.789244 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.789300 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 06:25:08.789608 3520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59558
I20260926 06:25:08.796815 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:08.790544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51322 (local address 127.2.39.254:38215)
0926 06:25:08.790605 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:08.790607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:08.790646 (+ 39us) server_negotiation.cc:407] Connection header received
0926 06:25:08.790695 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:08.790696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:08.790706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:08.790775 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:08.791202 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.791953 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.792735 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:08.792971 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:08.793032 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:08.794280 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:08.794281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:08.794281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:08.794288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:08.795724 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.796075 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.796076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.796077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.796133 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:08.796332 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:08.796333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:08.796333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:08.796445 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:08.796542 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:08.796645 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 06:25:08.796717 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 06:25:08.797159 3320 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51322
I20260926 06:25:08.797802 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 3395
I20260926 06:25:08.804656 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 3535
I20260926 06:25:08.811003 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 3675
I20260926 06:25:08.816382 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 3291
2026-09-26T06:25:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2591 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3892](info): set up 2 sockets
Sep 26 06:25:08 dist-test-slave-fkbn krb5kdc[3892](info): commencing operation
krb5kdc: starting...
W20260926 06:25:09.953346 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:10.068919 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38823
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:38823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:10.147960 3905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:10.148150 3905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:10.148172 3905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:10.149317 3905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:10.149359 3905 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 06:25:10.149372 3905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:10.149408 3905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:10.149439 3905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:10.149456 3905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:10.150954 3905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:38823
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:10.151194 3905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:10.151425 3905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:10.156325 3911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:10.156309 3910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:10.156303 3913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:10.156684 3905 server_base.cc:1034] running on GCE node
I20260926 06:25:10.156973 3905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:10.157272 3905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:10.158442 3905 hybrid_clock.cc:648] HybridClock initialized: now 1790403910158412 us; error 46 us; skew 500 ppm
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:10.160954 3905 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:10.161329 3905 webserver.cc:460] Webserver started at http://127.2.39.254:42599/ using document root <none> and password file <none>
I20260926 06:25:10.161525 3905 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:10.161608 3905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:10.161726 3905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:10.163028 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "22498a95ef034d31a5a2dcb6e580f291"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "013176d9ce7830a7de3f1d8963b605c0"
server_key_iv: "b196b43bd397249f59be2f28c1a46f79"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.163487 3905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "22498a95ef034d31a5a2dcb6e580f291"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "013176d9ce7830a7de3f1d8963b605c0"
server_key_iv: "b196b43bd397249f59be2f28c1a46f79"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.164764 3905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:25:10.165647 3920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:10.165846 3905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:10.165946 3905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "22498a95ef034d31a5a2dcb6e580f291"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "013176d9ce7830a7de3f1d8963b605c0"
server_key_iv: "b196b43bd397249f59be2f28c1a46f79"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.166055 3905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:10.175685 3905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:10.177492 3905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:10.177665 3905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:10.182123 3905 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:38823
I20260926 06:25:10.182174 3982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:38823 every 8 connection(s)
I20260926 06:25:10.182476 3905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:10.183049 3983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:10.183267 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 3905
I20260926 06:25:10.183354 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:10.183550 2207 external_mini_cluster.cc:1428] Setting key 2b1b5cf3e4521a8df41537a3499c2fea
I20260926 06:25:10.185561 3983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291: Bootstrap starting.
I20260926 06:25:10.186192 3983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:10.186482 3983 log.cc:824] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:10.187135 3983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291: No bootstrap required, opened a new log
I20260926 06:25:10.189181 3983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22498a95ef034d31a5a2dcb6e580f291" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38823 } }
I20260926 06:25:10.189358 3983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:10.189404 3983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22498a95ef034d31a5a2dcb6e580f291, State: Initialized, Role: FOLLOWER
I20260926 06:25:10.189539 3983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22498a95ef034d31a5a2dcb6e580f291" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38823 } }
I20260926 06:25:10.189646 3983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:10.189685 3983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:10.189734 3983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:10.190450 3983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22498a95ef034d31a5a2dcb6e580f291" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38823 } }
I20260926 06:25:10.190567 3983 leader_election.cc:302] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22498a95ef034d31a5a2dcb6e580f291; no voters:
I20260926 06:25:10.190770 3983 leader_election.cc:288] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:10.190802 3988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:10.190915 3988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [term 1 LEADER]: Becoming Leader. State: Replica: 22498a95ef034d31a5a2dcb6e580f291, State: Running, Role: LEADER
I20260926 06:25:10.191008 3988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22498a95ef034d31a5a2dcb6e580f291" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38823 } }
I20260926 06:25:10.191043 3983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:10.191321 3990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22498a95ef034d31a5a2dcb6e580f291. Latest consensus state: current_term: 1 leader_uuid: "22498a95ef034d31a5a2dcb6e580f291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22498a95ef034d31a5a2dcb6e580f291" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38823 } } }
I20260926 06:25:10.191424 3989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22498a95ef034d31a5a2dcb6e580f291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22498a95ef034d31a5a2dcb6e580f291" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38823 } } }
I20260926 06:25:10.191491 3990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:10.191504 3989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:10.191854 3997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:10.193231 3997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:10.194344 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.184039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.2.39.254:38823)
0926 06:25:10.184222 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.184231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.184264 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:25:10.184655 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.184661 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.184704 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.184845 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:10.185457 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.186524 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.187318 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.187921 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.188055 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.190514 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:10.190518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:10.190518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:10.190547 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:10.192625 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.193059 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.193062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.193062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.193177 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.193404 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.193405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.193405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.193730 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:10.193888 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.194000 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.194136 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260926 06:25:10.194801 3997 catalog_manager.cc:1347] Generated new cluster ID: 7a8219ba8cce464ca7bdeefc12630ce0
I20260926 06:25:10.194847 3997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:10.210727 3997 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:10.211464 3997 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:10.216675 3997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291: Generated new TSK 0
I20260926 06:25:10.216866 3997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:10.245980 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38823
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:10.325671 4009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:10.325873 4009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:10.325897 4009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:10.327033 4009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:10.327075 4009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:10.327096 4009 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 06:25:10.327108 4009 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:25:10.327150 4009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:10.328560 4009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.39.254:38823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:10.328789 4009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:10.329031 4009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:10.333989 4015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:10.334040 4014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:10.334007 4017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:10.334250 4009 server_base.cc:1034] running on GCE node
I20260926 06:25:10.334503 4009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:10.334736 4009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:10.335897 4009 hybrid_clock.cc:648] HybridClock initialized: now 1790403910335893 us; error 36 us; skew 500 ppm
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:10.338413 4009 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:10.338774 4009 webserver.cc:460] Webserver started at http://127.2.39.193:37449/ using document root <none> and password file <none>
I20260926 06:25:10.338943 4009 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:10.338984 4009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:10.339066 4009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:10.339998 4009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "79c3f873886f3f12a149be191da5eccb"
server_key_iv: "825d86ceb9cc174873eb7f4ef5cd85f6"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.340281 4009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "79c3f873886f3f12a149be191da5eccb"
server_key_iv: "825d86ceb9cc174873eb7f4ef5cd85f6"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.341452 4009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:10.342196 4024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:10.342355 4009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:10.342434 4009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "79c3f873886f3f12a149be191da5eccb"
server_key_iv: "825d86ceb9cc174873eb7f4ef5cd85f6"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.342484 4009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:10.372581 4009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:10.374392 4009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:10.374495 4009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:10.374923 4009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:10.374956 4009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:10.374977 4009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:10.374990 4009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:10.381826 4009 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:34485
I20260926 06:25:10.381949 4139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:34485 every 8 connection(s)
I20260926 06:25:10.382228 4009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:10.384846 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.375495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:58169 (local address 127.2.39.254:38823)
0926 06:25:10.375569 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.375571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.375773 (+ 202us) server_negotiation.cc:407] Connection header received
0926 06:25:10.376278 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.376280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.376294 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.376372 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:10.377243 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.378165 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.379206 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.379562 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.379636 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.381964 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:10.381966 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:10.381966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:10.381973 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:10.383490 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.383893 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.383894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.383895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.383953 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.384226 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.384227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.384227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.384346 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:10.384420 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.384680 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.384758 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:25:10.385035 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.375471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38823 (local address 127.2.39.193:58169)
0926 06:25:10.375633 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:10.375672 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:10.376139 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:10.376455 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:10.376459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:10.376826 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:10.377121 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.377126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.378276 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.378277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.379097 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.379100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.379656 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.379657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.379782 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.380270 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:10.380284 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:10.381862 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:10.383593 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.383599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.383601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.383816 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.384045 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.384045 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.384046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.384156 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.384413 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:10.384418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:10.384551 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 06:25:10.384659 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 06:25:10.384808 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260926 06:25:10.385481 4140 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38823
I20260926 06:25:10.385603 4140 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:10.385778 4140 heartbeater.cc:511] Master 127.2.39.254:38823 requested a full tablet report, sending...
I20260926 06:25:10.386252 3936 ts_manager.cc:194] Registered new tserver with Master: ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193:34485)
I20260926 06:25:10.386896 3936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:58169
I20260926 06:25:10.390743 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 4009
I20260926 06:25:10.390851 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:10.391057 2207 external_mini_cluster.cc:1428] Setting key 53e9d259a24515388b639433378fc6e1
I20260926 06:25:10.392136 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.386578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:33989 (local address 127.2.39.254:38823)
0926 06:25:10.386618 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.386620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.386744 (+ 124us) server_negotiation.cc:407] Connection header received
0926 06:25:10.386813 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.386814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.386823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.386877 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:10.387212 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.387666 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.388358 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.388565 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.388648 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.389492 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:10.389493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:10.389494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:10.389500 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:10.390987 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.391381 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.391382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.391382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.391461 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.391631 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.391631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.391632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.391748 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:10.391814 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.391968 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.392067 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:25:10.392171 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.386521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38823 (local address 127.2.39.193:33989)
0926 06:25:10.386671 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:10.386680 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:10.386756 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:10.386882 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:10.386882 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:10.387081 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:10.387128 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.387129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.387754 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.387755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.388262 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.388264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.388648 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.388649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.388719 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.389138 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:10.389142 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:10.389401 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:10.391074 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.391075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.391075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.391269 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.391520 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.391521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.391521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.391572 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.391813 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:10.391814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:10.391861 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:25:10.391951 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 06:25:10.392052 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:10.416620 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38823
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:10.496439 4149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:10.496628 4149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:10.496651 4149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:10.497753 4149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:10.497788 4149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:10.497808 4149 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 06:25:10.497820 4149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:25:10.497838 4149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:10.499177 4149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.39.254:38823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:10.499354 4149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:10.499521 4149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:10.504521 4155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:10.504547 4154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:10.504561 4157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:10.504588 4149 server_base.cc:1034] running on GCE node
I20260926 06:25:10.504787 4149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:10.504977 4149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:10.506131 4149 hybrid_clock.cc:648] HybridClock initialized: now 1790403910506114 us; error 31 us; skew 500 ppm
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:10.508589 4149 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:10.509035 4149 webserver.cc:460] Webserver started at http://127.2.39.194:43373/ using document root <none> and password file <none>
I20260926 06:25:10.509249 4149 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:10.509315 4149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:10.509513 4149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:10.510541 4149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "01a62b6fbe63420f95fb901b18d9f890"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "23d754d37d250ae4684fba866ab1b82d"
server_key_iv: "baa287084af46f2c1f97e0aa88dae662"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.510911 4149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "01a62b6fbe63420f95fb901b18d9f890"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "23d754d37d250ae4684fba866ab1b82d"
server_key_iv: "baa287084af46f2c1f97e0aa88dae662"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.512166 4149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:10.512939 4164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:10.513156 4149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:10.513268 4149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "01a62b6fbe63420f95fb901b18d9f890"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "23d754d37d250ae4684fba866ab1b82d"
server_key_iv: "baa287084af46f2c1f97e0aa88dae662"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.513406 4149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:10.526306 4149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:10.528666 4149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:10.528839 4149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:10.529402 4149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:10.529474 4149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:10.529508 4149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:10.529531 4149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:10.537781 4149 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:44569
I20260926 06:25:10.537873 4279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:44569 every 8 connection(s)
I20260926 06:25:10.538218 4149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:10.538681 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.529749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:57483 (local address 127.2.39.254:38823)
0926 06:25:10.529812 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.529814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.530027 (+ 213us) server_negotiation.cc:407] Connection header received
0926 06:25:10.530637 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.530639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.530656 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.530734 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:10.531453 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.532008 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.533026 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.533275 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.533361 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.535631 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:10.535632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:10.535633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:10.535640 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:10.537089 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.537588 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.537589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.537589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.537644 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.537937 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.537938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.537938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.538102 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:10.538195 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.538489 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.538591 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:25:10.538889 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.529742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38823 (local address 127.2.39.194:57483)
0926 06:25:10.529910 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:10.529942 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:10.530498 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:10.530803 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:10.530806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:10.531024 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:10.531286 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.531290 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.532113 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.532114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.532920 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.532922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.533376 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.533377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.533518 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.534054 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:10.534082 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:10.535536 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:10.537187 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.537193 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.537195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.537483 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.537726 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.537727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.537727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.537869 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.538189 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:10.538192 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:10.538327 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 06:25:10.538459 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 06:25:10.538642 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260926 06:25:10.539498 4280 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38823
I20260926 06:25:10.539574 4280 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:10.539742 4280 heartbeater.cc:511] Master 127.2.39.254:38823 requested a full tablet report, sending...
I20260926 06:25:10.540115 3936 ts_manager.cc:194] Registered new tserver with Master: 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569)
I20260926 06:25:10.540536 3936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:57483
I20260926 06:25:10.541326 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 4149
I20260926 06:25:10.541427 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:10.541661 2207 external_mini_cluster.cc:1428] Setting key 09fd7ef9570f20ce426590ac409b9207
I20260926 06:25:10.546653 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.540665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:42867 (local address 127.2.39.254:38823)
0926 06:25:10.540705 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.540707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.540715 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:10.540734 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.540734 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.540743 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.540791 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:10.541135 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.541810 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.542588 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.542911 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.542989 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.543954 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:10.543955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:10.543955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:10.543961 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:10.545463 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.545918 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.545919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.545920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.545977 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.546204 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.546205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.546205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.546331 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:10.546380 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.546543 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.546596 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:25:10.546783 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.540495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38823 (local address 127.2.39.194:42867)
0926 06:25:10.540619 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:10.540629 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:10.540673 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:10.540793 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:10.540794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:10.540991 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:10.541054 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.541056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.541918 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.541919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.542513 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.542514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.542991 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.542992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.543047 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.543460 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:10.543464 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:10.543852 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:10.545552 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.545553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.545554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.545801 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.546052 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.546053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.546054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.546143 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.546397 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:10.546398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:10.546448 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 06:25:10.546524 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 06:25:10.546634 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:10.568585 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38823
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:10.651117 4289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:10.651304 4289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:10.651327 4289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:10.652390 4289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:10.652423 4289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:10.652443 4289 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 06:25:10.652455 4289 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:25:10.652472 4289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:10.654124 4289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.39.254:38823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:10.654306 4289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:10.654491 4289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:10.659415 4297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:10.659373 4294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:10.659618 4289 server_base.cc:1034] running on GCE node
W20260926 06:25:10.659390 4295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:10.659894 4289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:10.660084 4289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:10.661232 4289 hybrid_clock.cc:648] HybridClock initialized: now 1790403910661214 us; error 38 us; skew 500 ppm
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:10.663713 4289 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:10.664129 4289 webserver.cc:460] Webserver started at http://127.2.39.195:43301/ using document root <none> and password file <none>
I20260926 06:25:10.664412 4289 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:10.664485 4289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:10.664634 4289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:10.665764 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "d376dd96832041a9b601d0a3789bb497"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "c5d3887ef0a87740ec3f2237142aa2e5"
server_key_iv: "48c2a43b45bfa671e9edf059a7146ad0"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.666112 4289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "d376dd96832041a9b601d0a3789bb497"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "c5d3887ef0a87740ec3f2237142aa2e5"
server_key_iv: "48c2a43b45bfa671e9edf059a7146ad0"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.667402 4289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:10.668103 4304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:10.668303 4289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:10.668433 4289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "d376dd96832041a9b601d0a3789bb497"
format_stamp: "Formatted at 2026-09-26 06:25:10 on dist-test-slave-fkbn"
server_key: "c5d3887ef0a87740ec3f2237142aa2e5"
server_key_iv: "48c2a43b45bfa671e9edf059a7146ad0"
server_key_version: "encryptionkey@0"
I20260926 06:25:10.668486 4289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:10.679420 4289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:10.681499 4289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:10.681702 4289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:10.682204 4289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:10.682238 4289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:10.682260 4289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:10.682272 4289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:10.689716 4289 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:39577
I20260926 06:25:10.689800 4419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:39577 every 8 connection(s)
I20260926 06:25:10.690097 4289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:10.692886 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.682791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:52787 (local address 127.2.39.254:38823)
0926 06:25:10.682888 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.682890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.683104 (+ 214us) server_negotiation.cc:407] Connection header received
0926 06:25:10.683752 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.683754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.683767 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.683843 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:10.684717 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.685608 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.686845 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.687159 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.687254 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.689608 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:10.689610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:10.689610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:10.689619 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:10.691373 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.691811 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.691812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.691812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.691872 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.692173 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.692174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.692175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.692342 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:10.692433 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.692709 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.692791 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 06:25:10.693082 4313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.682775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38823 (local address 127.2.39.195:52787)
0926 06:25:10.682966 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:10.683013 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:10.683606 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:10.683929 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:10.683935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:10.684308 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:10.684605 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.684611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.685704 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.685705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.686741 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.686759 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.687283 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.687284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.687429 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.687975 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:10.687983 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:10.689477 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:10.691470 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.691477 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.691479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.691733 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.691953 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.691953 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.691954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.692070 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.692424 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:10.692430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:10.692569 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 06:25:10.692686 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 06:25:10.692849 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260926 06:25:10.693183 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 4289
I20260926 06:25:10.693281 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:10.693495 2207 external_mini_cluster.cc:1428] Setting key eff9a254da825d6ac615081d3e0088cf
I20260926 06:25:10.693591 4420 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38823
I20260926 06:25:10.693780 4420 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:10.693982 4420 heartbeater.cc:511] Master 127.2.39.254:38823 requested a full tablet report, sending...
I20260926 06:25:10.694334 3936 ts_manager.cc:194] Registered new tserver with Master: d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577)
I20260926 06:25:10.694834 3936 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:52787
I20260926 06:25:10.699982 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.694670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:53227 (local address 127.2.39.254:38823)
0926 06:25:10.694702 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.694704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.695122 (+ 418us) server_negotiation.cc:407] Connection header received
0926 06:25:10.695189 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.695190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.695198 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.695239 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:10.695546 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.696045 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.696720 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.696887 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.696938 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.697948 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:10.697949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:10.697949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:10.697954 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:10.699099 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.699409 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.699410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.699410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.699456 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.699616 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.699617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.699617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.699704 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:10.699752 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.699870 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.699922 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:25:10.699991 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.694601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38823 (local address 127.2.39.195:53227)
0926 06:25:10.695032 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:10.695045 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:10.695123 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:10.695244 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:10.695245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:10.695413 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:10.695465 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.695467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.696132 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.696133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.696646 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.696648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.696976 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.696977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.697053 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.697391 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:10.697395 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:10.697874 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:10.699168 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.699169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.699169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.699331 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.699513 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:10.699513 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:10.699514 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:10.699563 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:10.699756 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:10.699756 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:10.699790 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 06:25:10.699847 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 06:25:10.699921 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":78,"threads_started":1}
I20260926 06:25:10.704989 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:25:10 dist-test-slave-fkbn krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:10.727319 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.719905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.2.39.254:38823)
0926 06:25:10.719969 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.719971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.719979 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:10.720006 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.720006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.720021 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.720085 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:10.720759 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.721287 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.722218 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.722415 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.722511 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.724894 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:10.724895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:10.724896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:10.724903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:10.726176 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.726539 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.726540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.726540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.726597 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:10.726798 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:10.726799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:10.726799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:10.726905 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:10.727012 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.727144 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.727216 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 06:25:10.728946 3936 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:10.730549 3936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:10.736899 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.733059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.2.39.195:39577)
0926 06:25:10.733258 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.733263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.733480 (+ 217us) server_negotiation.cc:407] Connection header received
0926 06:25:10.733586 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.733588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.733604 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.733679 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:10.733927 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.734609 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.735758 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.736485 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.736560 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.736614 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.736754 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.736794 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":105,"threads_started":1}
I20260926 06:25:10.736913 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.732902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:39577 (local address 127.0.0.1:55440)
0926 06:25:10.733419 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:10.733428 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:10.733481 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:10.733759 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:10.733761 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:10.733764 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:10.733826 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.733827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.734693 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.734694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.735667 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.735670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.736583 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.736584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.736656 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.736668 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 06:25:10.736738 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 06:25:10.736795 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":5,"thread_start_us":44,"threads_started":1}
I20260926 06:25:10.737236 4441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.733242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.2.39.194:44569)
0926 06:25:10.733418 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.733423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.733615 (+ 192us) server_negotiation.cc:407] Connection header received
0926 06:25:10.733656 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.733659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.733673 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.733737 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:10.733918 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.734621 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.736056 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.736558 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.736627 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.736683 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.737089 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.737131 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260926 06:25:10.736917 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.733123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:44569 (local address 127.0.0.1:33298)
0926 06:25:10.733458 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:10.733469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:10.733529 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:10.733796 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:10.733796 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:10.733797 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:10.733830 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.733831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.734739 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.734740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.735962 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.735964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.736662 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.736663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.736723 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.736727 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:10.736774 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 06:25:10.736817 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":29,"thread_start_us":85,"threads_started":1}
I20260926 06:25:10.737737 4354 tablet_service.cc:1467] Processing CreateTablet for tablet 830d1bb2672c4263b99d608434d37267 (DEFAULT_TABLE table=test-table [id=a360c98962074db6a7ad1cc692d9155f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:10.738039 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830d1bb2672c4263b99d608434d37267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:10.738062 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 830d1bb2672c4263b99d608434d37267 (DEFAULT_TABLE table=test-table [id=a360c98962074db6a7ad1cc692d9155f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:10.738385 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830d1bb2672c4263b99d608434d37267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:10.738548 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.732902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:34485 (local address 127.0.0.1:42630)
0926 06:25:10.733263 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:10.733308 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:10.733375 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:10.733820 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:10.733820 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:10.733822 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:10.733875 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.733876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.735965 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.735965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.737204 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:10.737207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.738297 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.738298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:10.738365 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.738369 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:10.738438 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:25:10.738476 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":67,"threads_started":1}
I20260926 06:25:10.738677 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:10.733293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42630 (local address 127.2.39.193:34485)
0926 06:25:10.733646 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:10.733650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:10.733659 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:25:10.733698 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:10.733700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:10.733717 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:10.733819 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:10.734719 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.735883 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.737298 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:10.738201 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:10.738279 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:10.738328 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:10.738453 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 06:25:10.738495 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":83,"threads_started":1}
I20260926 06:25:10.739315 4074 tablet_service.cc:1467] Processing CreateTablet for tablet 830d1bb2672c4263b99d608434d37267 (DEFAULT_TABLE table=test-table [id=a360c98962074db6a7ad1cc692d9155f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:10.739629 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 830d1bb2672c4263b99d608434d37267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:10.740908 4443 tablet_bootstrap.cc:492] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: Bootstrap starting.
I20260926 06:25:10.741750 4443 tablet_bootstrap.cc:654] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:10.741925 4444 tablet_bootstrap.cc:492] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: Bootstrap starting.
I20260926 06:25:10.742081 4443 log.cc:824] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:10.742491 4444 tablet_bootstrap.cc:654] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:10.742779 4444 log.cc:824] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:10.743332 4447 tablet_bootstrap.cc:492] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Bootstrap starting.
I20260926 06:25:10.743474 4444 tablet_bootstrap.cc:492] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: No bootstrap required, opened a new log
I20260926 06:25:10.743541 4444 ts_tablet_manager.cc:1397] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:10.743829 4443 tablet_bootstrap.cc:492] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: No bootstrap required, opened a new log
I20260926 06:25:10.743896 4443 ts_tablet_manager.cc:1397] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 06:25:10.745539 4444 raft_consensus.cc:348] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } }
I20260926 06:25:10.745745 4444 raft_consensus.cc:374] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:10.745781 4444 raft_consensus.cc:729] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a62b6fbe63420f95fb901b18d9f890, State: Initialized, Role: FOLLOWER
I20260926 06:25:10.745893 4444 consensus_queue.cc:260] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } }
I20260926 06:25:10.745834 4443 raft_consensus.cc:348] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } }
I20260926 06:25:10.745966 4443 raft_consensus.cc:374] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:10.746002 4443 raft_consensus.cc:729] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d376dd96832041a9b601d0a3789bb497, State: Initialized, Role: FOLLOWER
I20260926 06:25:10.746084 4447 tablet_bootstrap.cc:654] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:10.746106 4443 consensus_queue.cc:260] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } }
I20260926 06:25:10.746152 4444 ts_tablet_manager.cc:1428] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:25:10.746351 4443 ts_tablet_manager.cc:1428] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:25:10.746377 4420 heartbeater.cc:503] Master 127.2.39.254:38823 was elected leader, sending a full tablet report...
I20260926 06:25:10.746411 4447 log.cc:824] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:10.746466 4280 heartbeater.cc:503] Master 127.2.39.254:38823 was elected leader, sending a full tablet report...
I20260926 06:25:10.747256 4447 tablet_bootstrap.cc:492] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: No bootstrap required, opened a new log
I20260926 06:25:10.747335 4447 ts_tablet_manager.cc:1397] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 06:25:10.748868 4447 raft_consensus.cc:348] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } }
I20260926 06:25:10.749011 4447 raft_consensus.cc:374] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:10.749055 4447 raft_consensus.cc:729] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3d4ecd19c54fb38c5db2ec1c9f3435, State: Initialized, Role: FOLLOWER
I20260926 06:25:10.749161 4447 consensus_queue.cc:260] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } }
I20260926 06:25:10.749406 4447 ts_tablet_manager.cc:1428] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:25:10.749455 4140 heartbeater.cc:503] Master 127.2.39.254:38823 was elected leader, sending a full tablet report...
W20260926 06:25:10.788918 4281 tablet.cc:2369] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:25:10.882949 4141 tablet.cc:2369] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:25:10.941052 4421 tablet.cc:2369] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:25:11.055943 4448 raft_consensus.cc:482] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:11.056133 4448 raft_consensus.cc:504] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } }
I20260926 06:25:11.056562 4448 leader_election.cc:288] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577), ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193:34485)
I20260926 06:25:11.059562 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.056867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:57871 (local address 127.2.39.195:39577)
0926 06:25:11.056922 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.056924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.056950 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:25:11.057039 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.057040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.057051 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.057105 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:11.057262 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.057855 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.058869 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.059209 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.059299 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.059306 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.059460 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.059495 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:25:11.059563 4442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.056843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:39953 (local address 127.2.39.193:34485)
0926 06:25:11.056898 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.056900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.056983 (+ 83us) server_negotiation.cc:407] Connection header received
0926 06:25:11.057073 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.057074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.057091 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.057157 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:11.057304 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.057835 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.058856 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.059209 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.059290 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.059299 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.059450 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.059485 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 06:25:11.059589 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.056768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:34485 (local address 127.2.39.194:39953)
0926 06:25:11.056894 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:11.056905 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:11.056995 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:11.057159 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:11.057160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:11.057162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:11.057231 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.057233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.057912 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.057913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.058753 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.058755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.059287 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.059288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.059363 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.059366 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:11.059441 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:25:11.059481 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":45,"thread_start_us":58,"threads_started":1}
I20260926 06:25:11.059592 4453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.056736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:39577 (local address 127.2.39.194:57871)
0926 06:25:11.056887 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:11.056900 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:11.056969 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:11.057118 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:11.057118 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:11.057120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:11.057180 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.057181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.057937 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.057938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.058754 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.058756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.059300 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.059301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.059371 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.059380 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 06:25:11.059452 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 06:25:11.059496 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":30,"thread_start_us":69,"threads_started":1}
I20260926 06:25:11.059932 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830d1bb2672c4263b99d608434d37267" candidate_uuid: "01a62b6fbe63420f95fb901b18d9f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" is_pre_election: true
I20260926 06:25:11.060110 4094 raft_consensus.cc:2454] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a62b6fbe63420f95fb901b18d9f890 in term 0.
I20260926 06:25:11.060163 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830d1bb2672c4263b99d608434d37267" candidate_uuid: "01a62b6fbe63420f95fb901b18d9f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d376dd96832041a9b601d0a3789bb497" is_pre_election: true
I20260926 06:25:11.060320 4374 raft_consensus.cc:2454] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a62b6fbe63420f95fb901b18d9f890 in term 0.
I20260926 06:25:11.060355 4165 leader_election.cc:302] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a62b6fbe63420f95fb901b18d9f890, ec3d4ecd19c54fb38c5db2ec1c9f3435; no voters:
I20260926 06:25:11.060477 4448 raft_consensus.cc:2781] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:11.060539 4448 raft_consensus.cc:482] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:11.060566 4448 raft_consensus.cc:3037] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:11.061260 4448 raft_consensus.cc:504] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } }
I20260926 06:25:11.061394 4448 leader_election.cc:288] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [CANDIDATE]: Term 1 election: Requested vote from peers d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577), ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193:34485)
I20260926 06:25:11.061563 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830d1bb2672c4263b99d608434d37267" candidate_uuid: "01a62b6fbe63420f95fb901b18d9f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435"
I20260926 06:25:11.061620 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "830d1bb2672c4263b99d608434d37267" candidate_uuid: "01a62b6fbe63420f95fb901b18d9f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d376dd96832041a9b601d0a3789bb497"
I20260926 06:25:11.061677 4374 raft_consensus.cc:3037] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:11.061668 4094 raft_consensus.cc:3037] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:11.062507 4374 raft_consensus.cc:2454] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a62b6fbe63420f95fb901b18d9f890 in term 1.
I20260926 06:25:11.062507 4094 raft_consensus.cc:2454] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a62b6fbe63420f95fb901b18d9f890 in term 1.
I20260926 06:25:11.062672 4165 leader_election.cc:302] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a62b6fbe63420f95fb901b18d9f890, ec3d4ecd19c54fb38c5db2ec1c9f3435; no voters:
I20260926 06:25:11.062783 4448 raft_consensus.cc:2781] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:11.062880 4448 raft_consensus.cc:686] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 LEADER]: Becoming Leader. State: Replica: 01a62b6fbe63420f95fb901b18d9f890, State: Running, Role: LEADER
I20260926 06:25:11.062968 4448 consensus_queue.cc:237] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } }
I20260926 06:25:11.063743 3937 catalog_manager.cc:5495] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194). New cstate: current_term: 1 leader_uuid: "01a62b6fbe63420f95fb901b18d9f890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } health_report { overall_health: HEALTHY } } }
Sep 26 06:25:11 dist-test-slave-fkbn krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403910, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:11.121503 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.114338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38823 (local address 127.0.0.1:41470)
0926 06:25:11.114531 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:11.114542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:11.114606 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:11.114743 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:11.114743 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:11.115042 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:11.115100 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.115102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.115798 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.115800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.116412 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.116414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.116742 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.116743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.116782 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.117295 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:11.117299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:11.118864 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:11.120554 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:11.120560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:11.120562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:11.120739 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:11.120965 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:11.120966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:11.120966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:11.121017 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:11.121238 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:11.121243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:11.121280 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 06:25:11.121329 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:25:11.121399 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260926 06:25:11.121543 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.114474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41470 (local address 127.2.39.254:38823)
0926 06:25:11.114549 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.114551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.114602 (+ 51us) server_negotiation.cc:407] Connection header received
0926 06:25:11.114655 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.114656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.114666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.114727 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:11.115190 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.115715 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.116490 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.116671 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.116734 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.118943 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:11.118945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:11.118945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:11.118951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:11.120479 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:11.120828 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:11.120829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:11.120830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:11.120893 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:11.121084 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:11.121085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:11.121085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:11.121185 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:11.121240 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.121391 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.121449 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 06:25:11.123733 3937 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:41470:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 06:25:11.123903 3937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:11.125365 4074 tablet_service.cc:1467] Processing CreateTablet for tablet b69b9fc480154530b95ca92295763dfc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9faf0fac36b4c588c15bf184a57a57e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:25:11.125402 4214 tablet_service.cc:1467] Processing CreateTablet for tablet b69b9fc480154530b95ca92295763dfc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9faf0fac36b4c588c15bf184a57a57e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:25:11.125384 4354 tablet_service.cc:1467] Processing CreateTablet for tablet b69b9fc480154530b95ca92295763dfc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9faf0fac36b4c588c15bf184a57a57e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:25:11.125557 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69b9fc480154530b95ca92295763dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:11.125547 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69b9fc480154530b95ca92295763dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:11.125547 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b69b9fc480154530b95ca92295763dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:11.126811 4444 tablet_bootstrap.cc:492] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: Bootstrap starting.
I20260926 06:25:11.127014 4443 tablet_bootstrap.cc:492] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: Bootstrap starting.
I20260926 06:25:11.127017 4447 tablet_bootstrap.cc:492] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: Bootstrap starting.
I20260926 06:25:11.127243 4444 tablet_bootstrap.cc:654] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:11.127401 4447 tablet_bootstrap.cc:654] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:11.127615 4443 tablet_bootstrap.cc:654] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:11.128181 4447 tablet_bootstrap.cc:492] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: No bootstrap required, opened a new log
I20260926 06:25:11.128252 4447 ts_tablet_manager.cc:1397] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:11.128304 4444 tablet_bootstrap.cc:492] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: No bootstrap required, opened a new log
I20260926 06:25:11.128378 4444 ts_tablet_manager.cc:1397] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:11.128485 4447 raft_consensus.cc:348] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.128571 4447 raft_consensus.cc:374] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:11.128520 4444 raft_consensus.cc:348] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.128597 4447 raft_consensus.cc:729] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3d4ecd19c54fb38c5db2ec1c9f3435, State: Initialized, Role: FOLLOWER
I20260926 06:25:11.128598 4444 raft_consensus.cc:374] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:11.128619 4444 raft_consensus.cc:729] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a62b6fbe63420f95fb901b18d9f890, State: Initialized, Role: FOLLOWER
I20260926 06:25:11.128646 4447 consensus_queue.cc:260] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.128661 4444 consensus_queue.cc:260] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.128741 4444 ts_tablet_manager.cc:1428] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:11.128741 4447 ts_tablet_manager.cc:1428] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:11.128806 4451 tablet_replica.cc:439] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } }
I20260926 06:25:11.128906 4451 tablet_replica.cc:442] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:11.128891 4448 tablet_replica.cc:439] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } }
I20260926 06:25:11.128973 4448 tablet_replica.cc:442] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:11.129230 4443 tablet_bootstrap.cc:492] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: No bootstrap required, opened a new log
I20260926 06:25:11.129272 4443 ts_tablet_manager.cc:1397] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:11.129407 4443 raft_consensus.cc:348] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.129464 4443 raft_consensus.cc:374] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:11.129503 4443 raft_consensus.cc:729] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d376dd96832041a9b601d0a3789bb497, State: Initialized, Role: FOLLOWER
I20260926 06:25:11.129549 4443 consensus_queue.cc:260] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.129652 4443 ts_tablet_manager.cc:1428] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:11.129729 4449 tablet_replica.cc:439] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } }
I20260926 06:25:11.129805 4449 tablet_replica.cc:442] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:11.407275 4448 raft_consensus.cc:482] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:11.407439 4448 raft_consensus.cc:504] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.407635 4448 leader_election.cc:288] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193:34485), d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577)
I20260926 06:25:11.407869 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69b9fc480154530b95ca92295763dfc" candidate_uuid: "01a62b6fbe63420f95fb901b18d9f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d376dd96832041a9b601d0a3789bb497" is_pre_election: true
I20260926 06:25:11.407852 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69b9fc480154530b95ca92295763dfc" candidate_uuid: "01a62b6fbe63420f95fb901b18d9f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" is_pre_election: true
I20260926 06:25:11.407969 4094 raft_consensus.cc:2454] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a62b6fbe63420f95fb901b18d9f890 in term 0.
I20260926 06:25:11.407972 4374 raft_consensus.cc:2454] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01a62b6fbe63420f95fb901b18d9f890 in term 0.
I20260926 06:25:11.408123 4167 leader_election.cc:302] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a62b6fbe63420f95fb901b18d9f890, d376dd96832041a9b601d0a3789bb497; no voters:
I20260926 06:25:11.408218 4448 raft_consensus.cc:2781] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:11.408257 4448 raft_consensus.cc:482] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:11.408275 4448 raft_consensus.cc:3037] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:11.409008 4448 raft_consensus.cc:504] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.409152 4448 leader_election.cc:288] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [CANDIDATE]: Term 1 election: Requested vote from peers ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193:34485), d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577)
I20260926 06:25:11.409296 4094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69b9fc480154530b95ca92295763dfc" candidate_uuid: "01a62b6fbe63420f95fb901b18d9f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435"
I20260926 06:25:11.409394 4094 raft_consensus.cc:3037] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:11.409341 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b69b9fc480154530b95ca92295763dfc" candidate_uuid: "01a62b6fbe63420f95fb901b18d9f890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d376dd96832041a9b601d0a3789bb497"
I20260926 06:25:11.409621 4374 raft_consensus.cc:3037] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:11.410115 4094 raft_consensus.cc:2454] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a62b6fbe63420f95fb901b18d9f890 in term 1.
I20260926 06:25:11.410292 4165 leader_election.cc:302] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a62b6fbe63420f95fb901b18d9f890, ec3d4ecd19c54fb38c5db2ec1c9f3435; no voters:
I20260926 06:25:11.410391 4448 raft_consensus.cc:2781] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:11.410437 4448 raft_consensus.cc:686] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [term 1 LEADER]: Becoming Leader. State: Replica: 01a62b6fbe63420f95fb901b18d9f890, State: Running, Role: LEADER
I20260926 06:25:11.410486 4374 raft_consensus.cc:2454] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01a62b6fbe63420f95fb901b18d9f890 in term 1.
I20260926 06:25:11.410485 4448 consensus_queue.cc:237] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.410758 4454 tablet_replica.cc:439] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: TxnStatusTablet state changed. Reason: New leader 01a62b6fbe63420f95fb901b18d9f890. Latest consensus state: current_term: 1 leader_uuid: "01a62b6fbe63420f95fb901b18d9f890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } }
I20260926 06:25:11.410868 4454 tablet_replica.cc:442] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:25:11.411073 4466 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 06:25:11.411051 3937 catalog_manager.cc:5495] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194). New cstate: current_term: 1 leader_uuid: "01a62b6fbe63420f95fb901b18d9f890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:11.411195 4466 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 06:25:11.482244 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.479555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:44569 (local address 127.0.0.1:33312)
0926 06:25:11.479636 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:11.479649 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:11.479702 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:11.479942 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:11.479943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:11.479944 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:11.480001 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.480004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.480690 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.480691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.481274 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.481276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.481701 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.481702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.481764 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.481777 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:11.482058 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:11.482067 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 06:25:11.482115 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:25:11.482162 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 06:25:11.482254 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.479657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.2.39.194:44569)
0926 06:25:11.479792 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.479794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.479809 (+ 15us) server_negotiation.cc:407] Connection header received
0926 06:25:11.479838 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.479839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.479853 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.479936 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:11.480101 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.480605 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.481362 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.481613 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.481690 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.481890 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:11.481983 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:11.482079 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.482124 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.482144 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260926 06:25:11.484200 4374 raft_consensus.cc:1261] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Refusing update from remote peer 01a62b6fbe63420f95fb901b18d9f890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:11.484220 4094 raft_consensus.cc:1261] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Refusing update from remote peer 01a62b6fbe63420f95fb901b18d9f890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:11.484560 4454 consensus_queue.cc:1035] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:11.484688 4448 consensus_queue.cc:1035] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:11.485944 4451 tablet_replica.cc:439] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: TxnStatusTablet state changed. Reason: New leader 01a62b6fbe63420f95fb901b18d9f890. Latest consensus state: current_term: 1 leader_uuid: "01a62b6fbe63420f95fb901b18d9f890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } }
I20260926 06:25:11.486042 4451 tablet_replica.cc:442] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:11.486402 4449 tablet_replica.cc:439] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: TxnStatusTablet state changed. Reason: New leader 01a62b6fbe63420f95fb901b18d9f890. Latest consensus state: current_term: 1 leader_uuid: "01a62b6fbe63420f95fb901b18d9f890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } }
I20260926 06:25:11.486500 4449 tablet_replica.cc:442] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:11.487740 4448 tablet_replica.cc:439] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01a62b6fbe63420f95fb901b18d9f890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } }
I20260926 06:25:11.487825 4448 tablet_replica.cc:442] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:25:11.487782 4454 tablet_replica.cc:439] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01a62b6fbe63420f95fb901b18d9f890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } }
I20260926 06:25:11.487875 4454 tablet_replica.cc:442] T b69b9fc480154530b95ca92295763dfc P 01a62b6fbe63420f95fb901b18d9f890: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:25:11.488015 4470 mvcc.cc:205] Tried to move back new op lower bound from 7333494421436608512 to 7333494421137498112. Current Snapshot: MvccSnapshot[applied={T|T < 7333494421436608512}]
I20260926 06:25:11.488255 4451 tablet_replica.cc:439] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01a62b6fbe63420f95fb901b18d9f890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } }
I20260926 06:25:11.488335 4451 tablet_replica.cc:442] T b69b9fc480154530b95ca92295763dfc P ec3d4ecd19c54fb38c5db2ec1c9f3435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:11.488397 4449 tablet_replica.cc:439] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01a62b6fbe63420f95fb901b18d9f890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } } }
I20260926 06:25:11.488471 4449 tablet_replica.cc:442] T b69b9fc480154530b95ca92295763dfc P d376dd96832041a9b601d0a3789bb497: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:11.488771 4472 mvcc.cc:205] Tried to move back new op lower bound from 7333494421436608512 to 7333494421137498112. Current Snapshot: MvccSnapshot[applied={T|T < 7333494421436608512}]
I20260926 06:25:11.489141 4471 mvcc.cc:205] Tried to move back new op lower bound from 7333494421436608512 to 7333494421137498112. Current Snapshot: MvccSnapshot[applied={T|T < 7333494421436608512 or (T in {7333494421436608512})}]
I20260926 06:25:11.493916 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.491371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.2.39.194:44569)
0926 06:25:11.491405 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.491407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.491536 (+ 129us) server_negotiation.cc:407] Connection header received
0926 06:25:11.491599 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.491600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.491610 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.491658 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:11.491817 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.492393 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.493220 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.493424 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.493496 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.493684 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:11.493716 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:11.493763 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.493820 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.493838 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 06:25:11.499045 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.496436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:54937 (local address 127.2.39.194:44569)
0926 06:25:11.496475 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.496476 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.496559 (+ 83us) server_negotiation.cc:407] Connection header received
0926 06:25:11.496632 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.496633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.496640 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.496684 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:11.496824 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.497247 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.498329 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.498516 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.498565 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.498785 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:11.498808 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:11.498848 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.498934 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.498952 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":21}
I20260926 06:25:11.499140 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.496348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:44569 (local address 127.2.39.194:54937)
0926 06:25:11.496501 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:11.496512 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:11.496574 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:11.496691 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:11.496692 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:11.496693 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:11.496762 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.496763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.497328 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.497329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.498238 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.498241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.498596 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.498597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.498667 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.498681 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:11.498874 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:11.498876 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:11.498931 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:25:11.498981 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260926 06:25:11.501225 4094 raft_consensus.cc:1261] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Refusing update from remote peer 01a62b6fbe63420f95fb901b18d9f890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:11.501260 4374 raft_consensus.cc:1261] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Refusing update from remote peer 01a62b6fbe63420f95fb901b18d9f890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:11.501447 4476 consensus_queue.cc:1035] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:11.501595 4448 consensus_queue.cc:1035] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:11.512823 3937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:11.512935 3937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:11.513662 3937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291: Sending DeleteTablet for 3 replicas of tablet 830d1bb2672c4263b99d608434d37267
I20260926 06:25:11.513922 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 830d1bb2672c4263b99d608434d37267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:11 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:55440
I20260926 06:25:11.513955 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet 830d1bb2672c4263b99d608434d37267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:11 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:42630
I20260926 06:25:11.514225 4214 tablet_service.cc:1514] Processing DeleteTablet for tablet 830d1bb2672c4263b99d608434d37267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:11 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:33298
I20260926 06:25:11.514452 4487 tablet_replica.cc:330] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: stopping tablet replica
I20260926 06:25:11.514443 3937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 06:25:11.514564 4487 raft_consensus.cc:2229] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:11.514626 4487 raft_consensus.cc:2258] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 06:25:11.514621 3937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:11.515019 4487 ts_tablet_manager.cc:1905] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:11.515609 4074 tablet_service.cc:1467] Processing CreateTablet for tablet 7e67e82b6d0144cd9bd3f984100f5aba (DEFAULT_TABLE table=test-table [id=b683e310f97d47a28750a39ca1daed85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:11.515671 4354 tablet_service.cc:1467] Processing CreateTablet for tablet 7e67e82b6d0144cd9bd3f984100f5aba (DEFAULT_TABLE table=test-table [id=b683e310f97d47a28750a39ca1daed85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:11.515728 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e67e82b6d0144cd9bd3f984100f5aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:11.515756 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e67e82b6d0144cd9bd3f984100f5aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:11.516798 4489 tablet_replica.cc:330] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: stopping tablet replica
I20260926 06:25:11.516899 4489 raft_consensus.cc:2229] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:11.517002 4447 tablet_bootstrap.cc:492] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435: Bootstrap starting.
I20260926 06:25:11.517010 4443 tablet_bootstrap.cc:492] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497: Bootstrap starting.
I20260926 06:25:11.517400 4443 tablet_bootstrap.cc:654] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:11.517400 4447 tablet_bootstrap.cc:654] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:11.517654 4489 raft_consensus.cc:2258] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:11.517872 4487 ts_tablet_manager.cc:1918] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:11.517983 4487 log.cc:1196] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/830d1bb2672c4263b99d608434d37267
I20260926 06:25:11.517982 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 7e67e82b6d0144cd9bd3f984100f5aba (DEFAULT_TABLE table=test-table [id=b683e310f97d47a28750a39ca1daed85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:11.518081 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e67e82b6d0144cd9bd3f984100f5aba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:11.518216 4447 tablet_bootstrap.cc:492] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435: No bootstrap required, opened a new log
I20260926 06:25:11.518276 4447 ts_tablet_manager.cc:1397] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:11.518415 4447 raft_consensus.cc:348] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.518461 4487 ts_tablet_manager.cc:1939] T 830d1bb2672c4263b99d608434d37267 P d376dd96832041a9b601d0a3789bb497: Deleting consensus metadata
I20260926 06:25:11.518492 4447 raft_consensus.cc:374] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:11.518513 4447 raft_consensus.cc:729] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3d4ecd19c54fb38c5db2ec1c9f3435, State: Initialized, Role: FOLLOWER
I20260926 06:25:11.518558 4447 consensus_queue.cc:260] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.518654 4447 ts_tablet_manager.cc:1428] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:11.519443 4444 tablet_bootstrap.cc:492] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890: Bootstrap starting.
I20260926 06:25:11.519716 3923 catalog_manager.cc:4841] TS d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577): tablet 830d1bb2672c4263b99d608434d37267 (table test-table [id=a360c98962074db6a7ad1cc692d9155f]) successfully deleted
I20260926 06:25:11.519858 4444 tablet_bootstrap.cc:654] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:11.519963 4443 tablet_bootstrap.cc:492] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497: No bootstrap required, opened a new log
I20260926 06:25:11.520011 4443 ts_tablet_manager.cc:1397] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 06:25:11.520169 4443 raft_consensus.cc:348] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.520251 4443 raft_consensus.cc:374] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:11.520278 4443 raft_consensus.cc:729] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d376dd96832041a9b601d0a3789bb497, State: Initialized, Role: FOLLOWER
I20260926 06:25:11.520329 4443 consensus_queue.cc:260] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.520432 4443 ts_tablet_manager.cc:1428] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:11.520699 4444 tablet_bootstrap.cc:492] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890: No bootstrap required, opened a new log
I20260926 06:25:11.520750 4444 ts_tablet_manager.cc:1397] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:11.520890 4489 ts_tablet_manager.cc:1905] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:11.520888 4444 raft_consensus.cc:348] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.520987 4444 raft_consensus.cc:374] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:11.521013 4444 raft_consensus.cc:729] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a62b6fbe63420f95fb901b18d9f890, State: Initialized, Role: FOLLOWER
I20260926 06:25:11.521054 4444 consensus_queue.cc:260] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.521178 4444 ts_tablet_manager.cc:1428] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:11.522943 4489 ts_tablet_manager.cc:1918] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:11.523025 4489 log.cc:1196] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/830d1bb2672c4263b99d608434d37267
I20260926 06:25:11.523253 4489 ts_tablet_manager.cc:1939] T 830d1bb2672c4263b99d608434d37267 P 01a62b6fbe63420f95fb901b18d9f890: Deleting consensus metadata
I20260926 06:25:11.523612 3923 catalog_manager.cc:4841] TS 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569): tablet 830d1bb2672c4263b99d608434d37267 (table test-table [id=a360c98962074db6a7ad1cc692d9155f]) successfully deleted
I20260926 06:25:11.525679 4488 tablet_replica.cc:330] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: stopping tablet replica
I20260926 06:25:11.525771 4488 raft_consensus.cc:2229] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:11.525821 4488 raft_consensus.cc:2258] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:11.526162 4488 ts_tablet_manager.cc:1905] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:11.527704 4488 ts_tablet_manager.cc:1918] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:11.527762 4488 log.cc:1196] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/830d1bb2672c4263b99d608434d37267
I20260926 06:25:11.527997 4488 ts_tablet_manager.cc:1939] T 830d1bb2672c4263b99d608434d37267 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Deleting consensus metadata
I20260926 06:25:11.528445 3921 catalog_manager.cc:4841] TS ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193:34485): tablet 830d1bb2672c4263b99d608434d37267 (table test-table [id=a360c98962074db6a7ad1cc692d9155f]) successfully deleted
I20260926 06:25:11.554988 4451 raft_consensus.cc:482] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:11.555099 4451 raft_consensus.cc:504] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.555400 4451 leader_election.cc:288] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569), d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577)
I20260926 06:25:11.558799 4439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.555828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:60689 (local address 127.2.39.195:39577)
0926 06:25:11.555871 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.555873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.555994 (+ 121us) server_negotiation.cc:407] Connection header received
0926 06:25:11.556064 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.556065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.556074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.556129 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:11.556308 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.556996 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.558061 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.558464 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.558537 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.558546 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.558693 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.558730 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:25:11.558802 4494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.555751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:39577 (local address 127.2.39.193:60689)
0926 06:25:11.555923 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:11.555932 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:11.556006 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:11.556133 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:11.556134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:11.556136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:11.556228 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.556229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.557109 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.557110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.557968 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.557971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.558543 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.558544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.558618 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.558622 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:11.558679 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 06:25:11.558717 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":43,"thread_start_us":68,"threads_started":1}
I20260926 06:25:11.559127 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e67e82b6d0144cd9bd3f984100f5aba" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d376dd96832041a9b601d0a3789bb497" is_pre_election: true
I20260926 06:25:11.559212 4374 raft_consensus.cc:2454] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 0.
I20260926 06:25:11.559309 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.555640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:49203 (local address 127.2.39.194:44569)
0926 06:25:11.555681 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.555683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.555798 (+ 115us) server_negotiation.cc:407] Connection header received
0926 06:25:11.555941 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.555942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.555954 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.556013 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:11.556208 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.556943 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.558409 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.558956 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.559042 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.559053 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.559198 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.559244 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:25:11.559386 4027 leader_election.cc:302] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d376dd96832041a9b601d0a3789bb497, ec3d4ecd19c54fb38c5db2ec1c9f3435; no voters:
I20260926 06:25:11.559504 4493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.555594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:44569 (local address 127.2.39.193:49203)
0926 06:25:11.555706 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:11.555717 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:11.555828 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:11.556064 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:11.556065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:11.556067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:11.556138 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.556141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.557028 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.557029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.558317 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.558319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.559040 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.559041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.559111 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.559115 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:11.559378 (+ 263us) client_negotiation.cc:240] Negotiation successful
0926 06:25:11.559417 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":24,"thread_start_us":61,"threads_started":1}
I20260926 06:25:11.559510 4451 raft_consensus.cc:2781] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:11.559578 4451 raft_consensus.cc:482] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:11.559603 4451 raft_consensus.cc:3037] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:11.560343 4451 raft_consensus.cc:504] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.560467 4451 leader_election.cc:288] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 election: Requested vote from peers 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569), d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577)
I20260926 06:25:11.560698 4374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e67e82b6d0144cd9bd3f984100f5aba" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d376dd96832041a9b601d0a3789bb497"
I20260926 06:25:11.560775 4374 raft_consensus.cc:3037] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:11.560760 4234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e67e82b6d0144cd9bd3f984100f5aba" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a62b6fbe63420f95fb901b18d9f890" is_pre_election: true
I20260926 06:25:11.560896 4234 raft_consensus.cc:2454] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 0.
I20260926 06:25:11.560770 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e67e82b6d0144cd9bd3f984100f5aba" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a62b6fbe63420f95fb901b18d9f890"
I20260926 06:25:11.561040 4233 raft_consensus.cc:3037] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:11.561529 4374 raft_consensus.cc:2454] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 1.
I20260926 06:25:11.561712 4027 leader_election.cc:302] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d376dd96832041a9b601d0a3789bb497, ec3d4ecd19c54fb38c5db2ec1c9f3435; no voters:
I20260926 06:25:11.561839 4451 raft_consensus.cc:2781] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:11.561900 4233 raft_consensus.cc:2454] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 1.
I20260926 06:25:11.561968 4451 raft_consensus.cc:686] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 LEADER]: Becoming Leader. State: Replica: ec3d4ecd19c54fb38c5db2ec1c9f3435, State: Running, Role: LEADER
I20260926 06:25:11.562040 4451 consensus_queue.cc:237] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:11.562776 3936 catalog_manager.cc:5495] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:11.572248 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.569518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.2.39.193:34485)
0926 06:25:11.569678 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.569680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.569690 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:11.569728 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.569728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.569748 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.569872 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:11.569974 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.570482 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.571327 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.571555 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.571626 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.571828 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:11.571972 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:11.572085 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.572124 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.572144 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260926 06:25:11.577169 4469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.574748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:43197 (local address 127.2.39.194:44569)
0926 06:25:11.574788 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.574790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.574945 (+ 155us) server_negotiation.cc:407] Connection header received
0926 06:25:11.575007 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.575008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.575020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.575086 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:11.575213 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.575672 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.576436 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.576643 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.576719 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.576910 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:11.576947 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:11.577004 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.577068 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.577091 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:25:11.577193 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.574677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:44569 (local address 127.2.39.193:43197)
0926 06:25:11.574874 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:11.574883 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:11.574954 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:11.575087 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:11.575088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:11.575089 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:11.575145 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.575147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.575753 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.575754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.576346 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.576348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.576728 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.576729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.576798 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.576820 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:11.577005 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:11.577007 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:11.577057 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:25:11.577099 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":133,"threads_started":1}
I20260926 06:25:11.579450 4374 raft_consensus.cc:1261] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Refusing update from remote peer ec3d4ecd19c54fb38c5db2ec1c9f3435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:11.579488 4233 raft_consensus.cc:1261] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Refusing update from remote peer ec3d4ecd19c54fb38c5db2ec1c9f3435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:11.579741 4495 consensus_queue.cc:1035] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:11.579892 4451 consensus_queue.cc:1035] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:11.587564 4483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.584599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:34485 (local address 127.2.39.194:47591)
0926 06:25:11.584658 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:11.584669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:11.584708 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:11.584967 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:11.584968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:11.584969 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:11.585010 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.585011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.585850 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.585851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.586723 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:11.586725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.587188 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.587189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:11.587245 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.587251 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:11.587421 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:11.587422 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:25:11.587465 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 06:25:11.587496 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 06:25:11.587568 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:11.584773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:47591 (local address 127.2.39.193:34485)
0926 06:25:11.584848 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:11.584850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:11.584859 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:25:11.584879 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:11.584880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:11.584891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:11.584956 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:11.585101 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.585778 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.586807 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:11.587086 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:11.587151 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:11.587327 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:11.587361 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:11.587422 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:11.587468 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 06:25:11.587487 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 06:25:11.587864 4471 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 06:25:13.111155 3935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:13.111256 3935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:13.112136 3935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291: Sending DeleteTablet for 3 replicas of tablet 7e67e82b6d0144cd9bd3f984100f5aba
I20260926 06:25:13.112380 4214 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e67e82b6d0144cd9bd3f984100f5aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:13 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:33298
I20260926 06:25:13.112401 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e67e82b6d0144cd9bd3f984100f5aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:13 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:55440
I20260926 06:25:13.112406 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e67e82b6d0144cd9bd3f984100f5aba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:13 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:42630
I20260926 06:25:13.112584 4528 tablet_replica.cc:330] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890: stopping tablet replica
I20260926 06:25:13.112639 4528 raft_consensus.cc:2229] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:13.112681 4528 raft_consensus.cc:2258] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:13.112782 4529 tablet_replica.cc:330] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435: stopping tablet replica
I20260926 06:25:13.112833 4529 raft_consensus.cc:2229] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:13.112926 4529 raft_consensus.cc:2258] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:13.112939 4528 ts_tablet_manager.cc:1905] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:13.113025 4530 tablet_replica.cc:330] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497: stopping tablet replica
I20260926 06:25:13.113077 4530 raft_consensus.cc:2229] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:13.113113 4530 raft_consensus.cc:2258] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:13.113291 4529 ts_tablet_manager.cc:1905] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:13.114681 4528 ts_tablet_manager.cc:1918] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:25:13.114737 4528 log.cc:1196] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/7e67e82b6d0144cd9bd3f984100f5aba
I20260926 06:25:13.114923 4530 ts_tablet_manager.cc:1905] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:13.114966 4528 ts_tablet_manager.cc:1939] T 7e67e82b6d0144cd9bd3f984100f5aba P 01a62b6fbe63420f95fb901b18d9f890: Deleting consensus metadata
I20260926 06:25:13.114971 4529 ts_tablet_manager.cc:1918] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:25:13.115015 4529 log.cc:1196] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/7e67e82b6d0144cd9bd3f984100f5aba
I20260926 06:25:13.115393 3923 catalog_manager.cc:4841] TS 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569): tablet 7e67e82b6d0144cd9bd3f984100f5aba (table test-table [id=b683e310f97d47a28750a39ca1daed85]) successfully deleted
I20260926 06:25:13.115504 4529 ts_tablet_manager.cc:1939] T 7e67e82b6d0144cd9bd3f984100f5aba P ec3d4ecd19c54fb38c5db2ec1c9f3435: Deleting consensus metadata
I20260926 06:25:13.115792 3921 catalog_manager.cc:4841] TS ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193:34485): tablet 7e67e82b6d0144cd9bd3f984100f5aba (table test-table [id=b683e310f97d47a28750a39ca1daed85]) successfully deleted
I20260926 06:25:13.116443 4530 ts_tablet_manager.cc:1918] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:25:13.116490 4530 log.cc:1196] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/7e67e82b6d0144cd9bd3f984100f5aba
I20260926 06:25:13.116650 4530 ts_tablet_manager.cc:1939] T 7e67e82b6d0144cd9bd3f984100f5aba P d376dd96832041a9b601d0a3789bb497: Deleting consensus metadata
I20260926 06:25:13.117077 3923 catalog_manager.cc:4841] TS d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577): tablet 7e67e82b6d0144cd9bd3f984100f5aba (table test-table [id=b683e310f97d47a28750a39ca1daed85]) successfully deleted
Sep 26 06:25:13 dist-test-slave-fkbn krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403913, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:25:13 dist-test-slave-fkbn krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403913, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:13.137281 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:13.130602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.2.39.254:38823)
0926 06:25:13.130754 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:13.130756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:13.130764 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:13.130817 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:13.130817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:13.130828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:13.130885 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:13.131420 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:13.131920 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:13.132607 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:13.132788 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:13.132836 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:13.135112 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:13.135113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:13.135113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:13.135119 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:13.136330 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:13.136655 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:13.136656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:13.136656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:13.136708 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:13.136865 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:13.136865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:13.136866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:13.136960 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:13.137034 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:13.137124 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 06:25:13.137181 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260926 06:25:13.138759 3935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:13.138957 3935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:13.140344 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 87a50af80cf84f9ead5837e4f00679b0 (DEFAULT_TABLE table=test-table [id=bd53363828024d9dadb09ec6fc25a273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:13.140378 4354 tablet_service.cc:1467] Processing CreateTablet for tablet 87a50af80cf84f9ead5837e4f00679b0 (DEFAULT_TABLE table=test-table [id=bd53363828024d9dadb09ec6fc25a273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:13.140475 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a50af80cf84f9ead5837e4f00679b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:13.140475 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a50af80cf84f9ead5837e4f00679b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:13.141652 4074 tablet_service.cc:1467] Processing CreateTablet for tablet 87a50af80cf84f9ead5837e4f00679b0 (DEFAULT_TABLE table=test-table [id=bd53363828024d9dadb09ec6fc25a273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:13.141785 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87a50af80cf84f9ead5837e4f00679b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:13.141855 4539 tablet_bootstrap.cc:492] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890: Bootstrap starting.
I20260926 06:25:13.142162 4539 tablet_bootstrap.cc:654] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:13.142745 4539 tablet_bootstrap.cc:492] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890: No bootstrap required, opened a new log
I20260926 06:25:13.142807 4539 ts_tablet_manager.cc:1397] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:13.142930 4539 raft_consensus.cc:348] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.143002 4539 raft_consensus.cc:374] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:13.143023 4539 raft_consensus.cc:729] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a62b6fbe63420f95fb901b18d9f890, State: Initialized, Role: FOLLOWER
I20260926 06:25:13.143054 4541 tablet_bootstrap.cc:492] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Bootstrap starting.
I20260926 06:25:13.143090 4539 consensus_queue.cc:260] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.143421 4541 tablet_bootstrap.cc:654] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:13.143604 4542 tablet_bootstrap.cc:492] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497: Bootstrap starting.
I20260926 06:25:13.143625 4539 ts_tablet_manager.cc:1428] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:13.144021 4542 tablet_bootstrap.cc:654] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:13.144104 4541 tablet_bootstrap.cc:492] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435: No bootstrap required, opened a new log
I20260926 06:25:13.144158 4541 ts_tablet_manager.cc:1397] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:13.144291 4541 raft_consensus.cc:348] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.144363 4541 raft_consensus.cc:374] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:13.144387 4541 raft_consensus.cc:729] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3d4ecd19c54fb38c5db2ec1c9f3435, State: Initialized, Role: FOLLOWER
I20260926 06:25:13.144444 4541 consensus_queue.cc:260] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.144557 4541 ts_tablet_manager.cc:1428] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:13.144641 4542 tablet_bootstrap.cc:492] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497: No bootstrap required, opened a new log
I20260926 06:25:13.144690 4542 ts_tablet_manager.cc:1397] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:13.144799 4542 raft_consensus.cc:348] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.144847 4542 raft_consensus.cc:374] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:13.144892 4542 raft_consensus.cc:729] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d376dd96832041a9b601d0a3789bb497, State: Initialized, Role: FOLLOWER
I20260926 06:25:13.144953 4542 consensus_queue.cc:260] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.145124 4542 ts_tablet_manager.cc:1428] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:13.367149 4518 raft_consensus.cc:482] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:13.367283 4518 raft_consensus.cc:504] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.367458 4518 leader_election.cc:288] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569), d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577)
I20260926 06:25:13.367655 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a50af80cf84f9ead5837e4f00679b0" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a62b6fbe63420f95fb901b18d9f890" is_pre_election: true
I20260926 06:25:13.367733 4373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a50af80cf84f9ead5837e4f00679b0" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d376dd96832041a9b601d0a3789bb497" is_pre_election: true
I20260926 06:25:13.367801 4233 raft_consensus.cc:2454] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 0.
I20260926 06:25:13.367836 4373 raft_consensus.cc:2454] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 0.
I20260926 06:25:13.367985 4027 leader_election.cc:302] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a62b6fbe63420f95fb901b18d9f890, ec3d4ecd19c54fb38c5db2ec1c9f3435; no voters:
I20260926 06:25:13.368098 4518 raft_consensus.cc:2781] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:13.368142 4518 raft_consensus.cc:482] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:13.368182 4518 raft_consensus.cc:3037] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:13.368981 4518 raft_consensus.cc:504] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.369123 4518 leader_election.cc:288] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 election: Requested vote from peers 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569), d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577)
I20260926 06:25:13.369271 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a50af80cf84f9ead5837e4f00679b0" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a62b6fbe63420f95fb901b18d9f890"
I20260926 06:25:13.369308 4373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87a50af80cf84f9ead5837e4f00679b0" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d376dd96832041a9b601d0a3789bb497"
I20260926 06:25:13.369367 4233 raft_consensus.cc:3037] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:13.369379 4373 raft_consensus.cc:3037] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:13.370188 4233 raft_consensus.cc:2454] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 1.
I20260926 06:25:13.370187 4373 raft_consensus.cc:2454] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 1.
I20260926 06:25:13.370394 4027 leader_election.cc:302] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a62b6fbe63420f95fb901b18d9f890, ec3d4ecd19c54fb38c5db2ec1c9f3435; no voters:
I20260926 06:25:13.370481 4518 raft_consensus.cc:2781] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:13.370537 4518 raft_consensus.cc:686] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 LEADER]: Becoming Leader. State: Replica: ec3d4ecd19c54fb38c5db2ec1c9f3435, State: Running, Role: LEADER
I20260926 06:25:13.370599 4518 consensus_queue.cc:237] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.371289 3937 catalog_manager.cc:5495] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:13.428403 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:13.426149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.2.39.193:34485)
0926 06:25:13.426298 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:13.426300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:13.426307 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:13.426325 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:13.426326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:13.426335 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:13.426398 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:13.426528 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:13.427007 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:13.427705 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:13.427906 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:13.427964 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:13.428097 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:13.428124 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:13.428163 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:13.428208 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 06:25:13.428225 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260926 06:25:13.430588 4373 raft_consensus.cc:1261] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Refusing update from remote peer ec3d4ecd19c54fb38c5db2ec1c9f3435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:13.430733 4233 raft_consensus.cc:1261] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Refusing update from remote peer ec3d4ecd19c54fb38c5db2ec1c9f3435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:13.430879 4522 consensus_queue.cc:1035] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:13.431111 4522 consensus_queue.cc:1035] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:13.440104 3937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:13.440184 3937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:13.440673 3937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291: Sending DeleteTablet for 3 replicas of tablet 87a50af80cf84f9ead5837e4f00679b0
I20260926 06:25:13.440887 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet 87a50af80cf84f9ead5837e4f00679b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:13 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:42630
I20260926 06:25:13.440939 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 87a50af80cf84f9ead5837e4f00679b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:13 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:55440
I20260926 06:25:13.440969 4529 tablet_replica.cc:330] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435: stopping tablet replica
I20260926 06:25:13.441035 4529 raft_consensus.cc:2229] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:13.441080 4529 raft_consensus.cc:2258] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:13.441074 4530 tablet_replica.cc:330] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497: stopping tablet replica
I20260926 06:25:13.441128 4530 raft_consensus.cc:2229] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:13.441162 4530 raft_consensus.cc:2258] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:13.441154 3937 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 06:25:13.441260 4214 tablet_service.cc:1514] Processing DeleteTablet for tablet 87a50af80cf84f9ead5837e4f00679b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:13 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:33298
W20260926 06:25:13.441310 3937 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:13.441347 4528 tablet_replica.cc:330] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890: stopping tablet replica
I20260926 06:25:13.441387 4528 raft_consensus.cc:2229] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:13.441421 4528 raft_consensus.cc:2258] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:13.441430 4529 ts_tablet_manager.cc:1905] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:13.441864 4528 ts_tablet_manager.cc:1905] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:13.442564 4354 tablet_service.cc:1467] Processing CreateTablet for tablet 7011854c05d54d7da81bfe8c64b07f94 (DEFAULT_TABLE table=test-table [id=8dc6b4c4cd344baea890ac87e890fe73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:13.442658 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7011854c05d54d7da81bfe8c64b07f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:13.443142 4074 tablet_service.cc:1467] Processing CreateTablet for tablet 7011854c05d54d7da81bfe8c64b07f94 (DEFAULT_TABLE table=test-table [id=8dc6b4c4cd344baea890ac87e890fe73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:13.443297 4528 ts_tablet_manager.cc:1918] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:13.443315 4214 tablet_service.cc:1467] Processing CreateTablet for tablet 7011854c05d54d7da81bfe8c64b07f94 (DEFAULT_TABLE table=test-table [id=8dc6b4c4cd344baea890ac87e890fe73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:13.443354 4528 log.cc:1196] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/87a50af80cf84f9ead5837e4f00679b0
I20260926 06:25:13.443363 4530 ts_tablet_manager.cc:1905] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:13.443404 4214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7011854c05d54d7da81bfe8c64b07f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:13.443578 4528 ts_tablet_manager.cc:1939] T 87a50af80cf84f9ead5837e4f00679b0 P 01a62b6fbe63420f95fb901b18d9f890: Deleting consensus metadata
I20260926 06:25:13.444597 4539 tablet_bootstrap.cc:492] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890: Bootstrap starting.
I20260926 06:25:13.444921 4539 tablet_bootstrap.cc:654] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:13.445077 4530 ts_tablet_manager.cc:1918] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:13.445145 4530 log.cc:1196] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/87a50af80cf84f9ead5837e4f00679b0
I20260926 06:25:13.445376 4530 ts_tablet_manager.cc:1939] T 87a50af80cf84f9ead5837e4f00679b0 P d376dd96832041a9b601d0a3789bb497: Deleting consensus metadata
I20260926 06:25:13.445477 4539 tablet_bootstrap.cc:492] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890: No bootstrap required, opened a new log
I20260926 06:25:13.445523 4539 ts_tablet_manager.cc:1397] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:13.445521 4542 tablet_bootstrap.cc:492] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497: Bootstrap starting.
I20260926 06:25:13.445673 3923 catalog_manager.cc:4841] TS 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569): tablet 87a50af80cf84f9ead5837e4f00679b0 (table test-table [id=bd53363828024d9dadb09ec6fc25a273]) successfully deleted
I20260926 06:25:13.445652 4539 raft_consensus.cc:348] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.445719 4539 raft_consensus.cc:374] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:13.445739 4539 raft_consensus.cc:729] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a62b6fbe63420f95fb901b18d9f890, State: Initialized, Role: FOLLOWER
I20260926 06:25:13.445794 4539 consensus_queue.cc:260] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.445880 4539 ts_tablet_manager.cc:1428] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:13.446000 3923 catalog_manager.cc:4841] TS d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577): tablet 87a50af80cf84f9ead5837e4f00679b0 (table test-table [id=bd53363828024d9dadb09ec6fc25a273]) successfully deleted
I20260926 06:25:13.446069 4542 tablet_bootstrap.cc:654] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:13.446483 4529 ts_tablet_manager.cc:1918] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:13.446483 4074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7011854c05d54d7da81bfe8c64b07f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:13.446553 4529 log.cc:1196] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/87a50af80cf84f9ead5837e4f00679b0
I20260926 06:25:13.446676 4542 tablet_bootstrap.cc:492] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497: No bootstrap required, opened a new log
I20260926 06:25:13.446707 4542 ts_tablet_manager.cc:1397] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:13.446759 4529 ts_tablet_manager.cc:1939] T 87a50af80cf84f9ead5837e4f00679b0 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Deleting consensus metadata
I20260926 06:25:13.446803 4542 raft_consensus.cc:348] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.446843 4542 raft_consensus.cc:374] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:13.446856 4542 raft_consensus.cc:729] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d376dd96832041a9b601d0a3789bb497, State: Initialized, Role: FOLLOWER
I20260926 06:25:13.446883 4542 consensus_queue.cc:260] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.446970 4542 ts_tablet_manager.cc:1428] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:13.447072 3921 catalog_manager.cc:4841] TS ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193:34485): tablet 87a50af80cf84f9ead5837e4f00679b0 (table test-table [id=bd53363828024d9dadb09ec6fc25a273]) successfully deleted
I20260926 06:25:13.447724 4541 tablet_bootstrap.cc:492] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Bootstrap starting.
I20260926 06:25:13.448026 4541 tablet_bootstrap.cc:654] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:13.448559 4541 tablet_bootstrap.cc:492] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435: No bootstrap required, opened a new log
I20260926 06:25:13.448623 4541 ts_tablet_manager.cc:1397] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 06:25:13.448740 4541 raft_consensus.cc:348] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.448807 4541 raft_consensus.cc:374] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:13.448828 4541 raft_consensus.cc:729] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3d4ecd19c54fb38c5db2ec1c9f3435, State: Initialized, Role: FOLLOWER
I20260926 06:25:13.448907 4541 consensus_queue.cc:260] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.448995 4541 ts_tablet_manager.cc:1428] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:13.554504 4522 raft_consensus.cc:482] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:13.554603 4522 raft_consensus.cc:504] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.554752 4522 leader_election.cc:288] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569), d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577)
I20260926 06:25:13.554975 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7011854c05d54d7da81bfe8c64b07f94" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a62b6fbe63420f95fb901b18d9f890" is_pre_election: true
I20260926 06:25:13.555095 4233 raft_consensus.cc:2454] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 0.
I20260926 06:25:13.555023 4373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7011854c05d54d7da81bfe8c64b07f94" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d376dd96832041a9b601d0a3789bb497" is_pre_election: true
I20260926 06:25:13.555152 4373 raft_consensus.cc:2454] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 0.
I20260926 06:25:13.555275 4027 leader_election.cc:302] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a62b6fbe63420f95fb901b18d9f890, ec3d4ecd19c54fb38c5db2ec1c9f3435; no voters:
I20260926 06:25:13.555411 4522 raft_consensus.cc:2781] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:13.555456 4522 raft_consensus.cc:482] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:13.555471 4522 raft_consensus.cc:3037] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:13.556185 4522 raft_consensus.cc:504] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.556327 4522 leader_election.cc:288] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 election: Requested vote from peers 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569), d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577)
I20260926 06:25:13.556490 4233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7011854c05d54d7da81bfe8c64b07f94" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a62b6fbe63420f95fb901b18d9f890"
I20260926 06:25:13.556568 4233 raft_consensus.cc:3037] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:13.556506 4373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7011854c05d54d7da81bfe8c64b07f94" candidate_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d376dd96832041a9b601d0a3789bb497"
I20260926 06:25:13.556608 4373 raft_consensus.cc:3037] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:13.557286 4233 raft_consensus.cc:2454] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 1.
I20260926 06:25:13.557428 4027 leader_election.cc:302] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01a62b6fbe63420f95fb901b18d9f890, ec3d4ecd19c54fb38c5db2ec1c9f3435; no voters:
I20260926 06:25:13.557471 4373 raft_consensus.cc:2454] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec3d4ecd19c54fb38c5db2ec1c9f3435 in term 1.
I20260926 06:25:13.557512 4522 raft_consensus.cc:2781] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:13.557547 4522 raft_consensus.cc:686] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [term 1 LEADER]: Becoming Leader. State: Replica: ec3d4ecd19c54fb38c5db2ec1c9f3435, State: Running, Role: LEADER
I20260926 06:25:13.557615 4522 consensus_queue.cc:237] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } }
I20260926 06:25:13.558175 3937 catalog_manager.cc:5495] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3d4ecd19c54fb38c5db2ec1c9f3435" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:13.591329 4233 raft_consensus.cc:1261] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Refusing update from remote peer ec3d4ecd19c54fb38c5db2ec1c9f3435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:13.591349 4374 raft_consensus.cc:1261] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Refusing update from remote peer ec3d4ecd19c54fb38c5db2ec1c9f3435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:13.591584 4522 consensus_queue.cc:1035] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a62b6fbe63420f95fb901b18d9f890" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:13.591735 4506 consensus_queue.cc:1035] T 7011854c05d54d7da81bfe8c64b07f94 P ec3d4ecd19c54fb38c5db2ec1c9f3435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d376dd96832041a9b601d0a3789bb497" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 06:25:13.599524 4510 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 06:25:15.117818 3935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:15.117939 3935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:15.118666 3935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22498a95ef034d31a5a2dcb6e580f291: Sending DeleteTablet for 3 replicas of tablet 7011854c05d54d7da81bfe8c64b07f94
I20260926 06:25:15.118948 4214 tablet_service.cc:1514] Processing DeleteTablet for tablet 7011854c05d54d7da81bfe8c64b07f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:15 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:33298
I20260926 06:25:15.118954 4074 tablet_service.cc:1514] Processing DeleteTablet for tablet 7011854c05d54d7da81bfe8c64b07f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:15 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:42630
I20260926 06:25:15.118994 4354 tablet_service.cc:1514] Processing DeleteTablet for tablet 7011854c05d54d7da81bfe8c64b07f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:15 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:55440
I20260926 06:25:15.119097 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 4009
I20260926 06:25:15.119153 4582 tablet_replica.cc:330] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497: stopping tablet replica
I20260926 06:25:15.119203 4582 raft_consensus.cc:2229] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:15.119235 4582 raft_consensus.cc:2258] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:15.119861 4582 ts_tablet_manager.cc:1905] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:15.120126 4580 tablet_replica.cc:330] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890: stopping tablet replica
I20260926 06:25:15.120232 4580 raft_consensus.cc:2229] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:15.120306 4580 raft_consensus.cc:2258] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:15.120577 4580 ts_tablet_manager.cc:1905] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:15.121621 4582 ts_tablet_manager.cc:1918] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:25:15.121686 4582 log.cc:1196] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/7011854c05d54d7da81bfe8c64b07f94
I20260926 06:25:15.121906 4582 ts_tablet_manager.cc:1939] T 7011854c05d54d7da81bfe8c64b07f94 P d376dd96832041a9b601d0a3789bb497: Deleting consensus metadata
I20260926 06:25:15.122308 3923 catalog_manager.cc:4841] TS d376dd96832041a9b601d0a3789bb497 (127.2.39.195:39577): tablet 7011854c05d54d7da81bfe8c64b07f94 (table test-table [id=8dc6b4c4cd344baea890ac87e890fe73]) successfully deleted
I20260926 06:25:15.122479 4580 ts_tablet_manager.cc:1918] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 06:25:15.122545 4580 log.cc:1196] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/7011854c05d54d7da81bfe8c64b07f94
I20260926 06:25:15.122763 4580 ts_tablet_manager.cc:1939] T 7011854c05d54d7da81bfe8c64b07f94 P 01a62b6fbe63420f95fb901b18d9f890: Deleting consensus metadata
I20260926 06:25:15.123231 3923 catalog_manager.cc:4841] TS 01a62b6fbe63420f95fb901b18d9f890 (127.2.39.194:44569): tablet 7011854c05d54d7da81bfe8c64b07f94 (table test-table [id=8dc6b4c4cd344baea890ac87e890fe73]) successfully deleted
W20260926 06:25:15.126087 3921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:34485 (error 108)
I20260926 06:25:15.126302 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 4149
I20260926 06:25:15.127023 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:15.126442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:34485 (local address 127.0.0.1:42662)
0926 06:25:15.126933 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:15.126980 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":56,"threads_started":1}
W20260926 06:25:15.127131 3921 catalog_manager.cc:4584] TS ec3d4ecd19c54fb38c5db2ec1c9f3435 (127.2.39.193:34485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7011854c05d54d7da81bfe8c64b07f94: Network error: Client connection negotiation failed: client connection to 127.2.39.193:34485: connect: Connection refused (error 111)
I20260926 06:25:15.132078 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 4289
I20260926 06:25:15.136555 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 3905
2026-09-26T06:25:15Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6319 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:15 dist-test-slave-fkbn krb5kdc[4587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:15 dist-test-slave-fkbn krb5kdc[4587](info): set up 2 sockets
Sep 26 06:25:15 dist-test-slave-fkbn krb5kdc[4587](info): commencing operation
krb5kdc: starting...
W20260926 06:25:16.169348 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:16 dist-test-slave-fkbn krb5kdc[4587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:16.288972 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:45673
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:45673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:16.369884 4600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:16.370100 4600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:16.370136 4600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:16.371265 4600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:16.371328 4600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:16.371371 4600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:16.371399 4600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:16.371421 4600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:16.372900 4600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:45673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:45673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:16.373168 4600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:16.373394 4600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:25:16.378618 4600 server_base.cc:1034] running on GCE node
W20260926 06:25:16.378599 4605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:16.378543 4608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:16.378545 4606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:16.379093 4600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:16.379441 4600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:16.380640 4600 hybrid_clock.cc:648] HybridClock initialized: now 1790403916380618 us; error 34 us; skew 500 ppm
Sep 26 06:25:16 dist-test-slave-fkbn krb5kdc[4587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:16.383090 4600 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:16.383476 4600 webserver.cc:460] Webserver started at http://127.2.39.254:34581/ using document root <none> and password file <none>
I20260926 06:25:16.383668 4600 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:16.383708 4600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:16.383793 4600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:16.384763 4600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "b2e1440635b549a399517b753698a442"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "1c5755a33565f5cd3194116a7fae0d11"
server_key_iv: "402713813e481143df322e55e4ac902b"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.385061 4600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "b2e1440635b549a399517b753698a442"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "1c5755a33565f5cd3194116a7fae0d11"
server_key_iv: "402713813e481143df322e55e4ac902b"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.386451 4600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:16.387135 4615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:16.387305 4600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:16.387392 4600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "b2e1440635b549a399517b753698a442"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "1c5755a33565f5cd3194116a7fae0d11"
server_key_iv: "402713813e481143df322e55e4ac902b"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.387513 4600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:16.430629 4600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:16.432695 4600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:16.432862 4600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:16.436583 4600 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:45673
I20260926 06:25:16.436625 4667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:45673 every 8 connection(s)
I20260926 06:25:16.436990 4600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:16.437916 4668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:16.440136 4668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442: Bootstrap starting.
I20260926 06:25:16.440716 4668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:16.441004 4668 log.cc:824] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:16.441694 4668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442: No bootstrap required, opened a new log
I20260926 06:25:16.443248 4668 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1440635b549a399517b753698a442" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45673 } }
I20260926 06:25:16.443387 4668 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:16.443408 4668 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2e1440635b549a399517b753698a442, State: Initialized, Role: FOLLOWER
I20260926 06:25:16.443574 4668 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1440635b549a399517b753698a442" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45673 } }
I20260926 06:25:16.443651 4668 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:16.443678 4668 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:16.443712 4668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:16.443943 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 4600
I20260926 06:25:16.444034 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:16.444213 2207 external_mini_cluster.cc:1428] Setting key 367d7f891f4fdfe71bbe3b405584273b
I20260926 06:25:16.444424 4668 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1440635b549a399517b753698a442" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45673 } }
I20260926 06:25:16.444509 4668 leader_election.cc:302] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2e1440635b549a399517b753698a442; no voters:
I20260926 06:25:16.444734 4668 leader_election.cc:288] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:16.444962 4671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:16.445029 4668 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:16.445137 4671 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [term 1 LEADER]: Becoming Leader. State: Replica: b2e1440635b549a399517b753698a442, State: Running, Role: LEADER
I20260926 06:25:16.445281 4671 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1440635b549a399517b753698a442" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45673 } }
I20260926 06:25:16.445659 4674 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2e1440635b549a399517b753698a442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1440635b549a399517b753698a442" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45673 } } }
I20260926 06:25:16.445665 4675 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2e1440635b549a399517b753698a442. Latest consensus state: current_term: 1 leader_uuid: "b2e1440635b549a399517b753698a442" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2e1440635b549a399517b753698a442" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45673 } } }
I20260926 06:25:16.445839 4674 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:16.445849 4675 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:16.446074 4682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:16.446666 4682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:16.448920 4682 catalog_manager.cc:1347] Generated new cluster ID: a2276a508e494b61855d78396f864940
I20260926 06:25:16.448976 4682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:25:16 dist-test-slave-fkbn krb5kdc[4587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:16.456220 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:16.444692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.2.39.254:45673)
0926 06:25:16.444898 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:16.444907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:16.445100 (+ 193us) server_negotiation.cc:407] Connection header received
0926 06:25:16.445254 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:16.445262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:16.445319 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:16.446301 (+ 982us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:16.446408 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.447854 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.448868 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.449340 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.449463 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:16.451842 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:16.451846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:16.451846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:16.451857 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:16.453535 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:16.455023 (+ 1488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:16.455026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:16.455026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:16.455139 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:16.455362 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:16.455363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:16.455364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:16.455621 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:16.455808 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:16.455875 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 06:25:16.456013 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260926 06:25:16.470490 4682 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:16.471249 4682 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:16.481032 4682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2e1440635b549a399517b753698a442: Generated new TSK 0
I20260926 06:25:16.481256 4682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:16.512466 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45673
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:16.593829 4694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:16.594054 4694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:16.594089 4694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:16.595268 4694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:16.595329 4694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:16.595384 4694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:16.596814 4694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:16.597080 4694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:16.597312 4694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:16.602339 4702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:16.602417 4700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:16.602516 4694 server_base.cc:1034] running on GCE node
W20260926 06:25:16.602348 4699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:16.602892 4694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:16.603161 4694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:16.604348 4694 hybrid_clock.cc:648] HybridClock initialized: now 1790403916604336 us; error 31 us; skew 500 ppm
Sep 26 06:25:16 dist-test-slave-fkbn krb5kdc[4587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:16.606976 4694 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:16.607342 4694 webserver.cc:460] Webserver started at http://127.2.39.193:34765/ using document root <none> and password file <none>
I20260926 06:25:16.607532 4694 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:16.607576 4694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:16.607666 4694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:16.608614 4694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "4ce5d6641a3e4d598674ea94c9096ce6"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "b77aa5dc00687c9291c826694100f873"
server_key_iv: "65efa205e7d5cdf278f1d712ca8fac5d"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.608907 4694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "4ce5d6641a3e4d598674ea94c9096ce6"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "b77aa5dc00687c9291c826694100f873"
server_key_iv: "65efa205e7d5cdf278f1d712ca8fac5d"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.610203 4694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:25:16.610849 4709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:16.611064 4694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:16.611143 4694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "4ce5d6641a3e4d598674ea94c9096ce6"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "b77aa5dc00687c9291c826694100f873"
server_key_iv: "65efa205e7d5cdf278f1d712ca8fac5d"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.611196 4694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:16.629941 4694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:16.632237 4694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:16.632359 4694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:16.632529 4694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:16.632845 4694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:16.632880 4694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:16.632902 4694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:16.632954 4694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:16.638681 4694 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:37769
I20260926 06:25:16.638731 4822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:37769 every 8 connection(s)
I20260926 06:25:16.639050 4694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:25:16 dist-test-slave-fkbn krb5kdc[4587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:16.647652 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 4694
I20260926 06:25:16.647761 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:16.647840 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:16.639699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:44951 (local address 127.2.39.254:45673)
0926 06:25:16.639738 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:16.639740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:16.639962 (+ 222us) server_negotiation.cc:407] Connection header received
0926 06:25:16.640421 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:16.640422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:16.640434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:16.640486 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:16.641364 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.641882 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.642791 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.642974 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.643030 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:16.645224 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:16.645225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:16.645226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:16.645231 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:16.646574 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:16.646972 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:16.646974 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:16.646974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:16.647048 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:16.647260 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:16.647260 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:16.647261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:16.647354 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:16.647402 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:16.647642 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 06:25:16.647726 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:25:16.647946 2207 external_mini_cluster.cc:1428] Setting key 9d508ff62a4256b8bbe20c436b2ad259
I20260926 06:25:16.648005 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:16.639688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:45673 (local address 127.2.39.193:44951)
0926 06:25:16.639837 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:16.639871 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:16.640329 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:16.640541 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:16.640544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:16.641105 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:16.641288 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:16.641292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.641951 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.641952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:16.642720 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:16.642721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.643060 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.643062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:16.643176 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:16.643733 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:16.643741 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:16.645152 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:16.646641 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:16.646646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:16.646648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:16.646888 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:16.647107 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:16.647108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:16.647108 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:16.647203 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:16.647406 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:16.647410 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:16.647514 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 06:25:16.647592 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 06:25:16.647698 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260926 06:25:16.648566 4823 heartbeater.cc:347] Connected to a master server at 127.2.39.254:45673
I20260926 06:25:16.648665 4823 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:16.648851 4823 heartbeater.cc:511] Master 127.2.39.254:45673 requested a full tablet report, sending...
I20260926 06:25:16.649379 4632 ts_manager.cc:194] Registered new tserver with Master: 4ce5d6641a3e4d598674ea94c9096ce6 (127.2.39.193:37769)
I20260926 06:25:16.650228 4632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:44951
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:16.672529 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45673
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:16.751387 4828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:16.751577 4828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:16.751598 4828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:16.753247 4828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:16.753289 4828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:16.753316 4828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:16.754664 4828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:16.754834 4828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:16.754997 4828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:16.760149 4834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:16.760187 4828 server_base.cc:1034] running on GCE node
W20260926 06:25:16.760205 4833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:16.760161 4836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:16.760485 4828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:16.760726 4828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:16.761883 4828 hybrid_clock.cc:648] HybridClock initialized: now 1790403916761889 us; error 46 us; skew 500 ppm
Sep 26 06:25:16 dist-test-slave-fkbn krb5kdc[4587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:16.764218 4828 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:16.764678 4828 webserver.cc:460] Webserver started at http://127.2.39.194:38587/ using document root <none> and password file <none>
I20260926 06:25:16.764895 4828 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:16.764962 4828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:16.765117 4828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:16.766188 4828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "12b361c66f7a4f45aeabd49ac7697758"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "a2de9d4d72c53293fee2ee4a270652b6"
server_key_iv: "bd47e0b45f3e122895259318e5a49407"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.766580 4828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "12b361c66f7a4f45aeabd49ac7697758"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "a2de9d4d72c53293fee2ee4a270652b6"
server_key_iv: "bd47e0b45f3e122895259318e5a49407"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.767987 4828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:16.768744 4843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:16.768932 4828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:16.769027 4828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "12b361c66f7a4f45aeabd49ac7697758"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "a2de9d4d72c53293fee2ee4a270652b6"
server_key_iv: "bd47e0b45f3e122895259318e5a49407"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.769110 4828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:16.781315 4828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:16.783200 4828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:16.783355 4828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:16.783591 4828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:16.783885 4828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:16.783941 4828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:16.783977 4828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:16.784005 4828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:16.789901 4828 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:44849
I20260926 06:25:16.789942 4956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:44849 every 8 connection(s)
I20260926 06:25:16.790256 4828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:25:16 dist-test-slave-fkbn krb5kdc[4587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:16.796996 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 4828
I20260926 06:25:16.797119 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:16.797343 2207 external_mini_cluster.cc:1428] Setting key 88f4b76758ef18b9d4c8c4600d2c789c
I20260926 06:25:16.799552 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:16.790994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:58803 (local address 127.2.39.254:45673)
0926 06:25:16.791080 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:16.791083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:16.791257 (+ 174us) server_negotiation.cc:407] Connection header received
0926 06:25:16.791727 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:16.791728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:16.791737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:16.791787 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:16.792692 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.793170 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.794257 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.794436 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.794484 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:16.796564 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:16.796566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:16.796566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:16.796571 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:16.798196 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:16.798570 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:16.798571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:16.798571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:16.798651 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:16.798931 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:16.798932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:16.798933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:16.799070 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:16.799138 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:16.799382 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 06:25:16.799468 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 06:25:16.799716 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:16.790957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:45673 (local address 127.2.39.194:58803)
0926 06:25:16.791136 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:16.791171 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:16.791644 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:16.791844 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:16.791847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:16.792422 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:16.792613 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:16.792618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.793253 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.793254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:16.794171 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:16.794174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.794520 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.794521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:16.794626 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:16.795155 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:16.795168 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:16.796490 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:16.798270 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:16.798275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:16.798276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:16.798497 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:16.798719 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:16.798720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:16.798721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:16.798844 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:16.799134 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:16.799138 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:16.799241 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 06:25:16.799349 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 06:25:16.799493 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260926 06:25:16.800305 4957 heartbeater.cc:347] Connected to a master server at 127.2.39.254:45673
I20260926 06:25:16.800432 4957 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:16.800596 4957 heartbeater.cc:511] Master 127.2.39.254:45673 requested a full tablet report, sending...
I20260926 06:25:16.800999 4632 ts_manager.cc:194] Registered new tserver with Master: 12b361c66f7a4f45aeabd49ac7697758 (127.2.39.194:44849)
I20260926 06:25:16.801530 4632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:58803
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:16.822697 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45673
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:16.901398 4962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:16.901613 4962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:16.901638 4962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:16.902787 4962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:16.902829 4962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:16.902855 4962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:16.904224 4962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:16.904445 4962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:16.904687 4962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:25:16.909664 4962 server_base.cc:1034] running on GCE node
W20260926 06:25:16.909639 4967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:16.909646 4968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:16.909627 4970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:16.910065 4962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:16.910287 4962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:16.911438 4962 hybrid_clock.cc:648] HybridClock initialized: now 1790403916911424 us; error 26 us; skew 500 ppm
Sep 26 06:25:16 dist-test-slave-fkbn krb5kdc[4587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:16.913892 4962 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:16.914364 4962 webserver.cc:460] Webserver started at http://127.2.39.195:39209/ using document root <none> and password file <none>
I20260926 06:25:16.914587 4962 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:16.914630 4962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:16.914768 4962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:16.915881 4962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "b500db1d5697494faa1426ffe2f4fff9"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "706a8b2da61392634bde44034ddbe09a"
server_key_iv: "aced8a4a5e9cf6940e12975f23e2a283"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.916244 4962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "b500db1d5697494faa1426ffe2f4fff9"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "706a8b2da61392634bde44034ddbe09a"
server_key_iv: "aced8a4a5e9cf6940e12975f23e2a283"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.917550 4962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:16.918336 4977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:16.918546 4962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:16.918643 4962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "b500db1d5697494faa1426ffe2f4fff9"
format_stamp: "Formatted at 2026-09-26 06:25:16 on dist-test-slave-fkbn"
server_key: "706a8b2da61392634bde44034ddbe09a"
server_key_iv: "aced8a4a5e9cf6940e12975f23e2a283"
server_key_version: "encryptionkey@0"
I20260926 06:25:16.918726 4962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:16.938459 4962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:16.940462 4962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:16.940690 4962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:16.940913 4962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:16.941224 4962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:16.941280 4962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:16.941323 4962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:16.941351 4962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:16.947286 4962 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:34203
I20260926 06:25:16.947322 5090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:34203 every 8 connection(s)
I20260926 06:25:16.947671 4962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:25:16 dist-test-slave-fkbn krb5kdc[4587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403916, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:16.957022 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:16.948429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:56071 (local address 127.2.39.254:45673)
0926 06:25:16.948490 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:16.948491 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:16.948707 (+ 216us) server_negotiation.cc:407] Connection header received
0926 06:25:16.949238 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:16.949239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:16.949251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:16.949307 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:16.950259 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.950759 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.951795 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.951984 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.952064 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:16.954156 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:16.954157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:16.954158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:16.954165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:16.955543 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:16.955948 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:16.955949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:16.955949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:16.956011 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:16.956416 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:16.956418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:16.956418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:16.956583 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:16.956663 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:16.956900 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 06:25:16.956960 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:25:16.957188 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 4962
I20260926 06:25:16.957245 5093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:16.948407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:45673 (local address 127.2.39.195:56071)
0926 06:25:16.948579 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:16.948618 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:16.949131 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:16.949383 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:16.949387 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:16.949966 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:16.950158 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:16.950162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.950850 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.950851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:16.951690 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:16.951692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:16.952068 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:16.952069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:16.952193 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:16.952680 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:16.952688 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:16.954046 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:16.955645 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:16.955651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:16.955653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:16.955868 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:16.956089 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:16.956090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:16.956090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:16.956240 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:16.956667 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:16.956672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:16.956784 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 06:25:16.956870 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 06:25:16.956982 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260926 06:25:16.957296 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:16.957461 2207 external_mini_cluster.cc:1428] Setting key 5a40a1078c39b84961f46e2967f1cab0
I20260926 06:25:16.957808 5091 heartbeater.cc:347] Connected to a master server at 127.2.39.254:45673
I20260926 06:25:16.957919 5091 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:16.958109 5091 heartbeater.cc:511] Master 127.2.39.254:45673 requested a full tablet report, sending...
I20260926 06:25:16.958456 4632 ts_manager.cc:194] Registered new tserver with Master: b500db1d5697494faa1426ffe2f4fff9 (127.2.39.195:34203)
I20260926 06:25:16.958832 4632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:56071
I20260926 06:25:16.969059 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 06:25:16.979676 4673 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:16.978712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.2.39.254:45673)
0926 06:25:16.978801 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:16.978803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:16.978905 (+ 102us) server_negotiation.cc:407] Connection header received
0926 06:25:16.978973 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:16.978974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:16.978986 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:16.979061 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:16.979604 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41758: BlockingRecv error: recv got EOF from 127.0.0.1:41758 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 06:25:16.980424 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 4694
I20260926 06:25:16.984594 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 4828
I20260926 06:25:16.988752 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 4962
I20260926 06:25:16.992908 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 4600
2026-09-26T06:25:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1855 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:17 dist-test-slave-fkbn krb5kdc[5103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:17 dist-test-slave-fkbn krb5kdc[5103](info): set up 2 sockets
Sep 26 06:25:17 dist-test-slave-fkbn krb5kdc[5103](info): commencing operation
krb5kdc: starting...
W20260926 06:25:18.041451 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:18 dist-test-slave-fkbn krb5kdc[5103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:18.160264 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:35323
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:35323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:18.240969 5116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:18.241192 5116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:18.241230 5116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:18.242377 5116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:18.242442 5116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:18.242465 5116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:18.242501 5116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:18.242524 5116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:18.243980 5116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:35323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:35323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:18.244244 5116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:18.244469 5116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:18.249207 5122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:18.249128 5121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:18.249133 5124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:18.249511 5116 server_base.cc:1034] running on GCE node
I20260926 06:25:18.249779 5116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:18.250087 5116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:18.251246 5116 hybrid_clock.cc:648] HybridClock initialized: now 1790403918251208 us; error 52 us; skew 500 ppm
Sep 26 06:25:18 dist-test-slave-fkbn krb5kdc[5103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403918, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:18.253756 5116 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:18.254174 5116 webserver.cc:460] Webserver started at http://127.2.39.254:33099/ using document root <none> and password file <none>
I20260926 06:25:18.254426 5116 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:18.254495 5116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:18.254638 5116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:18.255650 5116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "e75bb89af7c34c6c9e00d473094cba4e"
format_stamp: "Formatted at 2026-09-26 06:25:18 on dist-test-slave-fkbn"
server_key: "551dfa3ddeb2e4b6d03c16b0efa59f48"
server_key_iv: "1d564ccc425606ee5791b25251e2494d"
server_key_version: "encryptionkey@0"
I20260926 06:25:18.255991 5116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "e75bb89af7c34c6c9e00d473094cba4e"
format_stamp: "Formatted at 2026-09-26 06:25:18 on dist-test-slave-fkbn"
server_key: "551dfa3ddeb2e4b6d03c16b0efa59f48"
server_key_iv: "1d564ccc425606ee5791b25251e2494d"
server_key_version: "encryptionkey@0"
I20260926 06:25:18.257236 5116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:25:18.258010 5131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:18.258190 5116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:18.258277 5116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "e75bb89af7c34c6c9e00d473094cba4e"
format_stamp: "Formatted at 2026-09-26 06:25:18 on dist-test-slave-fkbn"
server_key: "551dfa3ddeb2e4b6d03c16b0efa59f48"
server_key_iv: "1d564ccc425606ee5791b25251e2494d"
server_key_version: "encryptionkey@0"
I20260926 06:25:18.258355 5116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:18.283746 5116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:18.285624 5116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:18.285795 5116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:18.289367 5116 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:35323
I20260926 06:25:18.289444 5183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:35323 every 8 connection(s)
I20260926 06:25:18.289808 5116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:18.290700 5184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:18.292956 5184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e: Bootstrap starting.
I20260926 06:25:18.293555 5184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:18.293937 5184 log.cc:824] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:18.294633 5184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e: No bootstrap required, opened a new log
I20260926 06:25:18.294879 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 5116
I20260926 06:25:18.294988 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:18.295167 2207 external_mini_cluster.cc:1428] Setting key 7f37d017f498ce9cfa163c9ac58fb562
I20260926 06:25:18.297011 5184 raft_consensus.cc:348] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75bb89af7c34c6c9e00d473094cba4e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35323 } }
I20260926 06:25:18.297176 5184 raft_consensus.cc:374] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:18.297214 5184 raft_consensus.cc:729] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75bb89af7c34c6c9e00d473094cba4e, State: Initialized, Role: FOLLOWER
I20260926 06:25:18.297395 5184 consensus_queue.cc:260] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75bb89af7c34c6c9e00d473094cba4e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35323 } }
I20260926 06:25:18.297488 5184 raft_consensus.cc:388] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:18.297521 5184 raft_consensus.cc:482] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:18.297593 5184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:18.298256 5184 raft_consensus.cc:504] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75bb89af7c34c6c9e00d473094cba4e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35323 } }
I20260926 06:25:18.298341 5184 leader_election.cc:302] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e75bb89af7c34c6c9e00d473094cba4e; no voters:
I20260926 06:25:18.298525 5184 leader_election.cc:288] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:18.298550 5189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:18.298672 5189 raft_consensus.cc:686] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [term 1 LEADER]: Becoming Leader. State: Replica: e75bb89af7c34c6c9e00d473094cba4e, State: Running, Role: LEADER
I20260926 06:25:18.298775 5184 sys_catalog.cc:564] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:18.298805 5189 consensus_queue.cc:237] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75bb89af7c34c6c9e00d473094cba4e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35323 } }
I20260926 06:25:18.299105 5189 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e75bb89af7c34c6c9e00d473094cba4e. Latest consensus state: current_term: 1 leader_uuid: "e75bb89af7c34c6c9e00d473094cba4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75bb89af7c34c6c9e00d473094cba4e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35323 } } }
I20260926 06:25:18.299175 5189 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:18.299132 5190 sys_catalog.cc:455] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e75bb89af7c34c6c9e00d473094cba4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e75bb89af7c34c6c9e00d473094cba4e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35323 } } }
I20260926 06:25:18.299331 5190 sys_catalog.cc:458] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e [sys.catalog]: This master's current role is: LEADER
W20260926 06:25:18.300551 5205 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:25:18.300626 5205 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:25:18.301085 5194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:18.301549 5194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:18.303000 5194 catalog_manager.cc:1347] Generated new cluster ID: 2049bafe142d41c5b0cfcb61b62d9429
I20260926 06:25:18.303046 5194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:25:18 dist-test-slave-fkbn krb5kdc[5103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:18.307168 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:18.295657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37050 (local address 127.2.39.254:35323)
0926 06:25:18.295837 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:18.295844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:18.295869 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:25:18.296992 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:18.296998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:18.297046 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:18.297156 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:18.298026 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:18.299297 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:18.300255 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:18.300848 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:18.300933 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:18.303568 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:18.303571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:18.303572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:18.303583 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:18.305460 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:18.305858 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:18.305861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:18.305861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:18.305976 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:18.306193 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:18.306195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:18.306195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:18.306492 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:18.306686 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:18.306776 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 06:25:18.306944 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260926 06:25:18.308068 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:25:18.315168 5194 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:18.315950 5194 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:18.326120 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:18.322023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.2.39.254:35323)
0926 06:25:18.322058 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:18.322060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:18.322204 (+ 144us) server_negotiation.cc:407] Connection header received
0926 06:25:18.322255 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:18.322256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:18.322280 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:18.322343 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:18.322727 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:18.323492 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:18.324512 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:18.324856 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:18.324928 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:18.325668 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:18.325670 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:18.325670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:18.325875 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:18.325893 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:18.325899 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:18.325962 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:18.326020 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 06:25:18.326046 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:25:18.328204 5194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e75bb89af7c34c6c9e00d473094cba4e: Generated new TSK 0
I20260926 06:25:18.328393 5194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:25:18.333005 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 5116
2026-09-26T06:25:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1338 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:18 dist-test-slave-fkbn krb5kdc[5217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:18 dist-test-slave-fkbn krb5kdc[5217](info): set up 2 sockets
Sep 26 06:25:18 dist-test-slave-fkbn krb5kdc[5217](info): commencing operation
krb5kdc: starting...
W20260926 06:25:19.373719 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:19 dist-test-slave-fkbn krb5kdc[5217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:19.490432 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:39369
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:39369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:19.569947 5230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:19.570184 5230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:19.570221 5230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:19.571363 5230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:19.571437 5230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:19.571463 5230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:19.571491 5230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:19.571532 5230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:19.573073 5230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:39369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:39369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:19.573343 5230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:19.573580 5230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:25:19.578485 5230 server_base.cc:1034] running on GCE node
W20260926 06:25:19.578341 5235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:19.578295 5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:19.578294 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:19.578886 5230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:19.579273 5230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:19.580426 5230 hybrid_clock.cc:648] HybridClock initialized: now 1790403919580403 us; error 43 us; skew 500 ppm
Sep 26 06:25:19 dist-test-slave-fkbn krb5kdc[5217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:19.582875 5230 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:19.583324 5230 webserver.cc:460] Webserver started at http://127.2.39.254:44189/ using document root <none> and password file <none>
I20260926 06:25:19.583557 5230 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:19.583602 5230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:19.583763 5230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:19.584761 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "ea8793ad785f48d69e05dae69e50fba2"
format_stamp: "Formatted at 2026-09-26 06:25:19 on dist-test-slave-fkbn"
server_key: "8a40805a77df89cf238bd38cbc4e0bf9"
server_key_iv: "517462ad108d894977ad78cfeed9374a"
server_key_version: "encryptionkey@0"
I20260926 06:25:19.585110 5230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "ea8793ad785f48d69e05dae69e50fba2"
format_stamp: "Formatted at 2026-09-26 06:25:19 on dist-test-slave-fkbn"
server_key: "8a40805a77df89cf238bd38cbc4e0bf9"
server_key_iv: "517462ad108d894977ad78cfeed9374a"
server_key_version: "encryptionkey@0"
I20260926 06:25:19.586444 5230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:19.587155 5245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:19.587328 5230 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:19.587396 5230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "ea8793ad785f48d69e05dae69e50fba2"
format_stamp: "Formatted at 2026-09-26 06:25:19 on dist-test-slave-fkbn"
server_key: "8a40805a77df89cf238bd38cbc4e0bf9"
server_key_iv: "517462ad108d894977ad78cfeed9374a"
server_key_version: "encryptionkey@0"
I20260926 06:25:19.587483 5230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:19.608731 5230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:19.610599 5230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:19.610752 5230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:19.614768 5230 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:39369
I20260926 06:25:19.614805 5297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:39369 every 8 connection(s)
I20260926 06:25:19.615152 5230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:19.616036 5298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:19.618754 5298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2: Bootstrap starting.
I20260926 06:25:19.619311 5298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:19.619586 5298 log.cc:824] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:19.620270 5298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2: No bootstrap required, opened a new log
I20260926 06:25:19.621852 5298 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8793ad785f48d69e05dae69e50fba2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39369 } }
I20260926 06:25:19.621984 5298 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:19.622006 5298 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea8793ad785f48d69e05dae69e50fba2, State: Initialized, Role: FOLLOWER
I20260926 06:25:19.622191 5298 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8793ad785f48d69e05dae69e50fba2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39369 } }
I20260926 06:25:19.622256 5298 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:19.622277 5298 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:19.622340 5298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:19.623117 5298 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8793ad785f48d69e05dae69e50fba2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39369 } }
I20260926 06:25:19.623221 5298 leader_election.cc:302] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea8793ad785f48d69e05dae69e50fba2; no voters:
I20260926 06:25:19.623394 5298 leader_election.cc:288] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:19.623446 5301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:19.623637 5298 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:19.623646 5301 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [term 1 LEADER]: Becoming Leader. State: Replica: ea8793ad785f48d69e05dae69e50fba2, State: Running, Role: LEADER
I20260926 06:25:19.623747 5301 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8793ad785f48d69e05dae69e50fba2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39369 } }
I20260926 06:25:19.624195 5303 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea8793ad785f48d69e05dae69e50fba2. Latest consensus state: current_term: 1 leader_uuid: "ea8793ad785f48d69e05dae69e50fba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8793ad785f48d69e05dae69e50fba2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39369 } } }
I20260926 06:25:19.624281 5303 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:19.624495 5310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:19.624178 5302 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea8793ad785f48d69e05dae69e50fba2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea8793ad785f48d69e05dae69e50fba2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39369 } } }
I20260926 06:25:19.624723 5302 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:19.624989 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 5230
I20260926 06:25:19.625002 5310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:19.625092 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:19.625288 2207 external_mini_cluster.cc:1428] Setting key a06aaa705df5a3e509a1f9a6966421d3
I20260926 06:25:19.626606 5310 catalog_manager.cc:1347] Generated new cluster ID: 5da6f87b550d4a8c92741a949dfe0e3e
I20260926 06:25:19.626647 5310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:25:19 dist-test-slave-fkbn krb5kdc[5217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:19.640575 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:19.625947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53414 (local address 127.2.39.254:39369)
0926 06:25:19.629773 (+ 3826us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:19.629782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:19.629813 (+ 31us) server_negotiation.cc:407] Connection header received
0926 06:25:19.629926 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:19.629934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:19.629992 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:19.630118 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:19.630888 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:19.632553 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:19.633674 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:19.634277 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:19.634384 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:19.637073 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:19.637077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:19.637077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:19.637094 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:19.639012 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:19.639395 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:19.639398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:19.639398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:19.639495 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:19.639711 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:19.639712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:19.639712 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:19.639987 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:19.640107 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:19.640232 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 06:25:19.640362 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3784,"thread_start_us":68,"threads_started":1}
I20260926 06:25:19.644474 5310 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:19.645136 5310 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:19.648412 5310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea8793ad785f48d69e05dae69e50fba2: Generated new TSK 0
I20260926 06:25:19.648561 5310 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:19.684958 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:39369
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:19.777730 5324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:19.777911 5324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:19.777933 5324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:19.779058 5324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:19.779098 5324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:19.779124 5324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:19.780541 5324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:19.780773 5324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:19.781000 5324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:19.786006 5332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:19.786042 5330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:19.786042 5329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:19.786098 5324 server_base.cc:1034] running on GCE node
I20260926 06:25:19.786417 5324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:19.786649 5324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:19.787806 5324 hybrid_clock.cc:648] HybridClock initialized: now 1790403919787790 us; error 45 us; skew 500 ppm
Sep 26 06:25:19 dist-test-slave-fkbn krb5kdc[5217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:19.790297 5324 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:19.790752 5324 webserver.cc:460] Webserver started at http://127.2.39.193:35305/ using document root <none> and password file <none>
I20260926 06:25:19.790985 5324 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:19.791052 5324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:19.791208 5324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:19.792204 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "a7c6e8d46009459db22d902e27128f43"
format_stamp: "Formatted at 2026-09-26 06:25:19 on dist-test-slave-fkbn"
server_key: "5135526e38133e4283abc0724436e376"
server_key_iv: "87d7e11e210a6a8fdbfbd82410591a01"
server_key_version: "encryptionkey@0"
I20260926 06:25:19.792564 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "a7c6e8d46009459db22d902e27128f43"
format_stamp: "Formatted at 2026-09-26 06:25:19 on dist-test-slave-fkbn"
server_key: "5135526e38133e4283abc0724436e376"
server_key_iv: "87d7e11e210a6a8fdbfbd82410591a01"
server_key_version: "encryptionkey@0"
I20260926 06:25:19.793879 5324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:19.794616 5339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:19.794790 5324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:19.794884 5324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "a7c6e8d46009459db22d902e27128f43"
format_stamp: "Formatted at 2026-09-26 06:25:19 on dist-test-slave-fkbn"
server_key: "5135526e38133e4283abc0724436e376"
server_key_iv: "87d7e11e210a6a8fdbfbd82410591a01"
server_key_version: "encryptionkey@0"
I20260926 06:25:19.794967 5324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:19.807658 5324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:19.809867 5324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:19.810021 5324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:19.810222 5324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:19.810516 5324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:19.810570 5324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:19.810611 5324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:19.810638 5324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:19.816332 5324 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:46257
I20260926 06:25:19.816390 5452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:46257 every 8 connection(s)
I20260926 06:25:19.816674 5324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:19.820091 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 5324
I20260926 06:25:19.820204 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:19.820392 2207 external_mini_cluster.cc:1428] Setting key 7b1f784412391468a981ea586e1cc95c
Sep 26 06:25:19 dist-test-slave-fkbn krb5kdc[5217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:19.826700 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:19.817441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:34157 (local address 127.2.39.254:39369)
0926 06:25:19.817505 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:19.817507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:19.817773 (+ 266us) server_negotiation.cc:407] Connection header received
0926 06:25:19.818234 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:19.818235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:19.818248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:19.818303 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:19.819229 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:19.819790 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:19.821146 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:19.821535 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:19.821626 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:19.823730 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:19.823731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:19.823732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:19.823740 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:19.825306 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:19.825767 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:19.825768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:19.825769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:19.825827 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:19.826071 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:19.826072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:19.826072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:19.826185 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:19.826263 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:19.826524 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 06:25:19.826610 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:25:19.826893 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:19.817437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:39369 (local address 127.2.39.193:34157)
0926 06:25:19.817635 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:19.817682 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:19.818141 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:19.818442 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:19.818446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:19.818964 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:19.819138 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:19.819142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:19.819895 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:19.819896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:19.821027 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:19.821029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:19.821648 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:19.821650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:19.821759 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:19.822242 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:19.822250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:19.823649 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:19.825396 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:19.825402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:19.825404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:19.825685 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:19.825891 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:19.825892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:19.825892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:19.826002 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:19.826253 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:19.826259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:19.826395 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 06:25:19.826502 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 06:25:19.826663 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260926 06:25:19.827378 5453 heartbeater.cc:347] Connected to a master server at 127.2.39.254:39369
I20260926 06:25:19.827454 5453 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:19.827641 5453 heartbeater.cc:511] Master 127.2.39.254:39369 requested a full tablet report, sending...
I20260926 06:25:19.828187 5262 ts_manager.cc:194] Registered new tserver with Master: a7c6e8d46009459db22d902e27128f43 (127.2.39.193:46257)
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
I20260926 06:25:19.828779 5262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:34157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:19.846383 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:39369
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:19.927354 5458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:19.927568 5458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:19.927593 5458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:19.928778 5458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:19.928815 5458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:19.928846 5458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:19.930459 5458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:19.930655 5458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:19.930850 5458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:19.936146 5466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:19.936120 5464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:19.936141 5463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:19.936499 5458 server_base.cc:1034] running on GCE node
I20260926 06:25:19.936692 5458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:19.936919 5458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:19.938074 5458 hybrid_clock.cc:648] HybridClock initialized: now 1790403919938038 us; error 58 us; skew 500 ppm
Sep 26 06:25:19 dist-test-slave-fkbn krb5kdc[5217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:19.940482 5458 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:19.940919 5458 webserver.cc:460] Webserver started at http://127.2.39.194:44113/ using document root <none> and password file <none>
I20260926 06:25:19.941105 5458 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:19.941146 5458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:19.941232 5458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:19.942255 5458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "3f8edbb8aa1f4cc0af0ba1d6f66de968"
format_stamp: "Formatted at 2026-09-26 06:25:19 on dist-test-slave-fkbn"
server_key: "c917d04f21b879ad712415b09620ef30"
server_key_iv: "2538c3e4b760c891643929b320b17498"
server_key_version: "encryptionkey@0"
I20260926 06:25:19.942559 5458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "3f8edbb8aa1f4cc0af0ba1d6f66de968"
format_stamp: "Formatted at 2026-09-26 06:25:19 on dist-test-slave-fkbn"
server_key: "c917d04f21b879ad712415b09620ef30"
server_key_iv: "2538c3e4b760c891643929b320b17498"
server_key_version: "encryptionkey@0"
I20260926 06:25:19.943783 5458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:19.944504 5473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:19.944725 5458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:19.944837 5458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "3f8edbb8aa1f4cc0af0ba1d6f66de968"
format_stamp: "Formatted at 2026-09-26 06:25:19 on dist-test-slave-fkbn"
server_key: "c917d04f21b879ad712415b09620ef30"
server_key_iv: "2538c3e4b760c891643929b320b17498"
server_key_version: "encryptionkey@0"
I20260926 06:25:19.944911 5458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:19.969015 5458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:19.970956 5458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:19.971103 5458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:19.971329 5458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:19.971653 5458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:19.971706 5458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:19.971741 5458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:19.971788 5458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:19.977412 5458 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:36729
I20260926 06:25:19.977449 5586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:36729 every 8 connection(s)
I20260926 06:25:19.977811 5458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:19.981231 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 5458
I20260926 06:25:19.981349 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:19.981539 2207 external_mini_cluster.cc:1428] Setting key e33dfa650b9253875b0e3f9abc0ac51a
Sep 26 06:25:19 dist-test-slave-fkbn krb5kdc[5217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403919, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:19.987154 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:19.978561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:56617 (local address 127.2.39.254:39369)
0926 06:25:19.978609 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:19.978612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:19.978817 (+ 205us) server_negotiation.cc:407] Connection header received
0926 06:25:19.979268 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:19.979269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:19.979278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:19.979329 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:19.980205 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:19.980692 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:19.982078 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:19.982276 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:19.982357 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:19.984432 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:19.984433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:19.984433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:19.984443 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:19.985813 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:19.986223 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:19.986225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:19.986225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:19.986283 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:19.986536 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:19.986537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:19.986538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:19.986648 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:19.986723 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:19.986989 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 06:25:19.987068 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:25:19.987372 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:19.978543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:39369 (local address 127.2.39.194:56617)
0926 06:25:19.978703 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:19.978735 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:19.979191 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:19.979382 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:19.979385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:19.979935 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:19.980133 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:19.980138 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:19.980773 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:19.980775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:19.981994 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:19.981996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:19.982370 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:19.982371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:19.982522 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:19.983010 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:19.983018 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:19.984329 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:19.985903 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:19.985910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:19.985912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:19.986132 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:19.986349 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:19.986350 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:19.986350 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:19.986463 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:19.986715 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:19.986721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:19.986855 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 06:25:19.986964 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 06:25:19.987113 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260926 06:25:19.987848 5587 heartbeater.cc:347] Connected to a master server at 127.2.39.254:39369
I20260926 06:25:19.987953 5587 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:19.988139 5587 heartbeater.cc:511] Master 127.2.39.254:39369 requested a full tablet report, sending...
I20260926 06:25:19.988519 5262 ts_manager.cc:194] Registered new tserver with Master: 3f8edbb8aa1f4cc0af0ba1d6f66de968 (127.2.39.194:36729)
I20260926 06:25:19.989018 5262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:56617
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:20.008656 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:39369
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:20.090502 5592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:20.090736 5592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:20.090758 5592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:20.091882 5592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:20.091925 5592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:20.091952 5592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:20.093333 5592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:20.093565 5592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:20.093861 5592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:25:20.098982 5592 server_base.cc:1034] running on GCE node
W20260926 06:25:20.098963 5600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:20.098978 5598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:20.099004 5597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:20.099467 5592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:20.099783 5592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:20.100996 5592 hybrid_clock.cc:648] HybridClock initialized: now 1790403920100971 us; error 33 us; skew 500 ppm
Sep 26 06:25:20 dist-test-slave-fkbn krb5kdc[5217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:20.103734 5592 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:20.104192 5592 webserver.cc:460] Webserver started at http://127.2.39.195:43225/ using document root <none> and password file <none>
I20260926 06:25:20.104451 5592 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:20.104526 5592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:20.104684 5592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:20.105865 5592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "99711aa71b9b4f7a959040348284e9c2"
format_stamp: "Formatted at 2026-09-26 06:25:20 on dist-test-slave-fkbn"
server_key: "7e9b4134be80fdaee8c93b50eae14c0a"
server_key_iv: "e5059287de1150880cd9b79d801cad87"
server_key_version: "encryptionkey@0"
I20260926 06:25:20.106240 5592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "99711aa71b9b4f7a959040348284e9c2"
format_stamp: "Formatted at 2026-09-26 06:25:20 on dist-test-slave-fkbn"
server_key: "7e9b4134be80fdaee8c93b50eae14c0a"
server_key_iv: "e5059287de1150880cd9b79d801cad87"
server_key_version: "encryptionkey@0"
I20260926 06:25:20.107616 5592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:20.108347 5607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:20.108511 5592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:20.108615 5592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "99711aa71b9b4f7a959040348284e9c2"
format_stamp: "Formatted at 2026-09-26 06:25:20 on dist-test-slave-fkbn"
server_key: "7e9b4134be80fdaee8c93b50eae14c0a"
server_key_iv: "e5059287de1150880cd9b79d801cad87"
server_key_version: "encryptionkey@0"
I20260926 06:25:20.108696 5592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:20.132270 5592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:20.134351 5592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:20.134511 5592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:20.134743 5592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:20.135056 5592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:20.135120 5592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:20.135162 5592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:20.135210 5592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:20.140784 5592 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:42447
I20260926 06:25:20.140856 5720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:42447 every 8 connection(s)
I20260926 06:25:20.141172 5592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:20.143723 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 5592
I20260926 06:25:20.143810 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:20.143975 2207 external_mini_cluster.cc:1428] Setting key 54b16b1e94aad784c2e3117ac0cb6620
Sep 26 06:25:20 dist-test-slave-fkbn krb5kdc[5217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403920, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:20.151399 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:20.141929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:35251 (local address 127.2.39.254:39369)
0926 06:25:20.141971 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:20.141973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:20.142180 (+ 207us) server_negotiation.cc:407] Connection header received
0926 06:25:20.142676 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:20.142677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:20.142686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:20.142743 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:20.143667 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:20.144444 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:20.145618 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:20.145815 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:20.145874 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:20.148255 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:20.148257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:20.148257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:20.148266 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:20.150058 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:20.150533 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:20.150534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:20.150534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:20.150592 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:20.150818 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:20.150818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:20.150819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:20.150945 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:20.151002 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:20.151247 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 06:25:20.151316 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:25:20.151607 5723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:20.141886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:39369 (local address 127.2.39.195:35251)
0926 06:25:20.142051 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:20.142091 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:20.142594 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:20.142795 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:20.142799 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:20.143365 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:20.143576 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:20.143582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:20.144538 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:20.144540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:20.145500 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:20.145502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:20.145890 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:20.145891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:20.145996 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:20.146578 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:20.146587 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:20.148146 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:20.150136 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:20.150142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:20.150144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:20.150454 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:20.150659 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:20.150660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:20.150660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:20.150760 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:20.151008 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:20.151012 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:20.151127 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 06:25:20.151215 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 06:25:20.151387 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260926 06:25:20.152153 5721 heartbeater.cc:347] Connected to a master server at 127.2.39.254:39369
I20260926 06:25:20.152267 5721 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:20.152468 5721 heartbeater.cc:511] Master 127.2.39.254:39369 requested a full tablet report, sending...
I20260926 06:25:20.152855 5262 ts_manager.cc:194] Registered new tserver with Master: 99711aa71b9b4f7a959040348284e9c2 (127.2.39.195:42447)
I20260926 06:25:20.153225 5262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:35251
I20260926 06:25:20.155661 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:25:20 dist-test-slave-fkbn krb5kdc[5217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403920, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 06:25:20 dist-test-slave-fkbn krb5kdc[5217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403920, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:20.178437 5320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:20.171049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.2.39.254:39369)
0926 06:25:20.171128 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:20.171131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:20.171248 (+ 117us) server_negotiation.cc:407] Connection header received
0926 06:25:20.171334 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:20.171335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:20.171351 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:20.171417 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:20.171970 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:20.172572 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:20.173360 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:20.173552 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:20.173670 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:20.175934 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:20.175935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:20.175935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:20.175943 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:20.177186 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:20.177608 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:20.177610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:20.177610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:20.177669 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:20.177864 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:20.177873 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:20.177873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:20.178015 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:20.178118 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:20.178257 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 06:25:20.178335 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 06:25:20.178710 5262 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53430
I20260926 06:25:20.179173 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 5324
I20260926 06:25:20.183564 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 5458
I20260926 06:25:20.187664 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 5592
I20260926 06:25:20.192106 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 5230
2026-09-26T06:25:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1860 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:20 dist-test-slave-fkbn krb5kdc[5733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:20 dist-test-slave-fkbn krb5kdc[5733](info): set up 2 sockets
Sep 26 06:25:20 dist-test-slave-fkbn krb5kdc[5733](info): commencing operation
krb5kdc: starting...
W20260926 06:25:21.233459 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:21 dist-test-slave-fkbn krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:21.349215 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:45271
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:45271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:21.428656 5746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:21.428845 5746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:21.428865 5746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:21.429984 5746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:21.430027 5746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:21.430042 5746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:21.430054 5746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:21.430066 5746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:21.431453 5746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:45271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:45271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:21.431622 5746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:21.431773 5746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:21.436786 5752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:21.436764 5751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:21.436756 5754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:21.436942 5746 server_base.cc:1034] running on GCE node
I20260926 06:25:21.437184 5746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:21.437454 5746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:21.438588 5746 hybrid_clock.cc:648] HybridClock initialized: now 1790403921438597 us; error 49 us; skew 500 ppm
Sep 26 06:25:21 dist-test-slave-fkbn krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:21.440974 5746 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:21.441419 5746 webserver.cc:460] Webserver started at http://127.2.39.254:40003/ using document root <none> and password file <none>
I20260926 06:25:21.441680 5746 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:21.441746 5746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:21.441964 5746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:21.443013 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "4e1cb915dbde434e82aa7126815f171d"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "ae9524f7660f6f7c2d8cb7ebce154de0"
server_key_iv: "f70996edf33844a42c8164323ddb39ba"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.443359 5746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "4e1cb915dbde434e82aa7126815f171d"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "ae9524f7660f6f7c2d8cb7ebce154de0"
server_key_iv: "f70996edf33844a42c8164323ddb39ba"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.444608 5746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:21.445312 5761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.445545 5746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:21.445675 5746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "4e1cb915dbde434e82aa7126815f171d"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "ae9524f7660f6f7c2d8cb7ebce154de0"
server_key_iv: "f70996edf33844a42c8164323ddb39ba"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.445755 5746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:21.464234 5746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:21.466286 5746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:21.466447 5746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:21.470458 5746 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:45271
I20260926 06:25:21.470530 5813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:45271 every 8 connection(s)
I20260926 06:25:21.470994 5746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:21.471686 5814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:21.473201 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 5746
I20260926 06:25:21.473321 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:21.473487 2207 external_mini_cluster.cc:1428] Setting key 84bf0edd4c25455607a69dc1e43f67ca
I20260926 06:25:21.474309 5814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d: Bootstrap starting.
I20260926 06:25:21.475006 5814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:21.475315 5814 log.cc:824] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:21.476130 5814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d: No bootstrap required, opened a new log
I20260926 06:25:21.478586 5814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1cb915dbde434e82aa7126815f171d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45271 } }
I20260926 06:25:21.478754 5814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:21.478794 5814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e1cb915dbde434e82aa7126815f171d, State: Initialized, Role: FOLLOWER
I20260926 06:25:21.478957 5814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1cb915dbde434e82aa7126815f171d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45271 } }
I20260926 06:25:21.479070 5814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:21.479102 5814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:21.479151 5814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:21.479698 5814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1cb915dbde434e82aa7126815f171d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45271 } }
I20260926 06:25:21.479782 5814 leader_election.cc:302] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e1cb915dbde434e82aa7126815f171d; no voters:
I20260926 06:25:21.479961 5814 leader_election.cc:288] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:21.480014 5819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:21.480234 5819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [term 1 LEADER]: Becoming Leader. State: Replica: 4e1cb915dbde434e82aa7126815f171d, State: Running, Role: LEADER
I20260926 06:25:21.480218 5814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:21.480443 5819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1cb915dbde434e82aa7126815f171d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45271 } }
I20260926 06:25:21.480825 5821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e1cb915dbde434e82aa7126815f171d. Latest consensus state: current_term: 1 leader_uuid: "4e1cb915dbde434e82aa7126815f171d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1cb915dbde434e82aa7126815f171d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45271 } } }
I20260926 06:25:21.480902 5821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [sys.catalog]: This master's current role is: LEADER
Sep 26 06:25:21 dist-test-slave-fkbn krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:21.481101 5820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e1cb915dbde434e82aa7126815f171d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e1cb915dbde434e82aa7126815f171d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45271 } } }
I20260926 06:25:21.481186 5820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:21.481634 5826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:21.482144 5826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:21.483765 5826 catalog_manager.cc:1347] Generated new cluster ID: 05b69c832fee4ce1945ab0b0bc5a8fd9
I20260926 06:25:21.483814 5826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:21.485133 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:21.474042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.2.39.254:45271)
0926 06:25:21.474221 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:21.474231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:21.474261 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:25:21.474813 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:21.474821 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:21.474881 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:21.475042 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:21.475808 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.477177 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.478070 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.478710 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.478820 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:21.481535 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:21.481539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:21.481539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:21.481550 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:21.483532 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:21.483940 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:21.483943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:21.483943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:21.484046 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:21.484260 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:21.484261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:21.484261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:21.484557 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:21.484715 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:21.484788 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 06:25:21.484940 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260926 06:25:21.515089 5826 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:21.515642 5826 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:21.523306 5826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e1cb915dbde434e82aa7126815f171d: Generated new TSK 0
I20260926 06:25:21.523502 5826 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:21.552327 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45271
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:21.635202 5840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:21.635386 5840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:21.635408 5840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:21.636600 5840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:21.636641 5840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:21.636667 5840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:21.638252 5840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:21.638465 5840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:21.638680 5840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:21.643678 5846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:21.643689 5845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:21.643821 5840 server_base.cc:1034] running on GCE node
W20260926 06:25:21.643985 5848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:21.644207 5840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:21.644392 5840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:21.645560 5840 hybrid_clock.cc:648] HybridClock initialized: now 1790403921645551 us; error 40 us; skew 500 ppm
Sep 26 06:25:21 dist-test-slave-fkbn krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:21.648110 5840 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:21.648586 5840 webserver.cc:460] Webserver started at http://127.2.39.193:39809/ using document root <none> and password file <none>
I20260926 06:25:21.648839 5840 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:21.648909 5840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:21.649065 5840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:21.650236 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "90e8787abec7445bba51b75d77367175"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "38bd02f849921521abe0849d327c7de2"
server_key_iv: "0775ba6de25aae67f965423e42855b8d"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.650595 5840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "90e8787abec7445bba51b75d77367175"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "38bd02f849921521abe0849d327c7de2"
server_key_iv: "0775ba6de25aae67f965423e42855b8d"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.651857 5840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:21.652833 5855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.653079 5840 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:21.653175 5840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "90e8787abec7445bba51b75d77367175"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "38bd02f849921521abe0849d327c7de2"
server_key_iv: "0775ba6de25aae67f965423e42855b8d"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.653254 5840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:21.664515 5840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:21.666415 5840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:21.666567 5840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:21.666770 5840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:21.667061 5840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:21.667114 5840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.667152 5840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:21.667181 5840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.673120 5840 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:45903
I20260926 06:25:21.673260 5968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:45903 every 8 connection(s)
I20260926 06:25:21.673534 5840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:21.676640 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 5840
I20260926 06:25:21.676760 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:21.676923 2207 external_mini_cluster.cc:1428] Setting key 129728d263b83f0b81caaeb7185657c8
Sep 26 06:25:21 dist-test-slave-fkbn krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:21.684641 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:21.674694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:53569 (local address 127.2.39.254:45271)
0926 06:25:21.674747 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:21.674748 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:21.674954 (+ 206us) server_negotiation.cc:407] Connection header received
0926 06:25:21.675890 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:21.675891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:21.675901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:21.675964 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:21.676555 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.677271 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.678861 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.679059 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.679171 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:21.681677 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:21.681679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:21.681679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:21.681685 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:21.683116 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:21.683613 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:21.683614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:21.683614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:21.683707 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:21.684012 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:21.684013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:21.684013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:21.684118 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:21.684172 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:21.684500 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 06:25:21.684577 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:25:21.684892 5971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:21.674665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:45271 (local address 127.2.39.193:53569)
0926 06:25:21.674853 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:21.674888 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:21.675751 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:21.676015 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:21.676018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:21.676248 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:21.676424 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:21.676428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.677354 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.677355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:21.678753 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:21.678756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.679182 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.679183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:21.679320 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:21.679885 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:21.679893 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:21.681600 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:21.683198 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:21.683204 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:21.683207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:21.683508 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:21.683792 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:21.683792 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:21.683793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:21.683936 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:21.684188 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:21.684193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:21.684354 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 06:25:21.684477 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 06:25:21.684650 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":125,"threads_started":1}
I20260926 06:25:21.685386 5969 heartbeater.cc:347] Connected to a master server at 127.2.39.254:45271
I20260926 06:25:21.685498 5969 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
I20260926 06:25:21.685710 5969 heartbeater.cc:511] Master 127.2.39.254:45271 requested a full tablet report, sending...
I20260926 06:25:21.686275 5778 ts_manager.cc:194] Registered new tserver with Master: 90e8787abec7445bba51b75d77367175 (127.2.39.193:45903)
I20260926 06:25:21.687031 5778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:53569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:21.705183 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45271
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:21.785295 5974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:21.785507 5974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:21.785543 5974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:21.786672 5974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:21.786738 5974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:21.786782 5974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:21.788230 5974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:21.788475 5974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:21.788689 5974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:25:21.793812 5974 server_base.cc:1034] running on GCE node
W20260926 06:25:21.793798 5980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:21.793828 5979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:21.793789 5982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:21.794175 5974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:21.794360 5974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:21.795492 5974 hybrid_clock.cc:648] HybridClock initialized: now 1790403921795479 us; error 30 us; skew 500 ppm
Sep 26 06:25:21 dist-test-slave-fkbn krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:21.797950 5974 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:21.798367 5974 webserver.cc:460] Webserver started at http://127.2.39.194:35605/ using document root <none> and password file <none>
I20260926 06:25:21.798596 5974 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:21.798640 5974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:21.798802 5974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:21.799795 5974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "9ac1c6ca61824c9ba18ea74f70f516c5"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "23a63eee2b355cd5d30288c3551af3bf"
server_key_iv: "f2a7f62cb3f04cc8a7457da7e6e7d61e"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.800124 5974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "9ac1c6ca61824c9ba18ea74f70f516c5"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "23a63eee2b355cd5d30288c3551af3bf"
server_key_iv: "f2a7f62cb3f04cc8a7457da7e6e7d61e"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.801337 5974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:25:21.802067 5989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.802232 5974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.802325 5974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "9ac1c6ca61824c9ba18ea74f70f516c5"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "23a63eee2b355cd5d30288c3551af3bf"
server_key_iv: "f2a7f62cb3f04cc8a7457da7e6e7d61e"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.802403 5974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:21.812918 5974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:21.815207 5974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:21.815387 5974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:21.815622 5974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:21.815904 5974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:21.815958 5974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.815995 5974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:21.816023 5974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.821376 5974 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:44283
I20260926 06:25:21.821451 6102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:44283 every 8 connection(s)
I20260926 06:25:21.821789 5974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:25:21 dist-test-slave-fkbn krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:21.829702 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 5974
I20260926 06:25:21.829813 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:21.829993 2207 external_mini_cluster.cc:1428] Setting key 098c14c4011f76fff928a2e97f30d995
I20260926 06:25:21.830320 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:21.822507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:50055 (local address 127.2.39.254:45271)
0926 06:25:21.822583 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:21.822585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:21.822726 (+ 141us) server_negotiation.cc:407] Connection header received
0926 06:25:21.823584 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:21.823585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:21.823594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:21.823645 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:21.824175 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.824662 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.825623 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.825808 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.825861 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:21.827806 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:21.827807 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:21.827808 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:21.827814 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:21.828927 (+ 1113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:21.829275 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:21.829276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:21.829276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:21.829321 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:21.829605 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:21.829606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:21.829607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:21.829758 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:21.829859 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:21.830153 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 06:25:21.830233 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:25:21.830497 6105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:21.822451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:45271 (local address 127.2.39.194:50055)
0926 06:25:21.822606 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:21.822640 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:21.823488 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:21.823695 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:21.823698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:21.823921 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:21.824095 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:21.824098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.824739 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.824739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:21.825497 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:21.825499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.825877 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.825878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:21.825976 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:21.826457 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:21.826465 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:21.827724 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:21.828995 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:21.829000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:21.829001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:21.829212 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:21.829376 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:21.829376 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:21.829376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:21.829466 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:21.829863 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:21.829869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:21.829994 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 06:25:21.830113 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:25:21.830263 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260926 06:25:21.830935 6103 heartbeater.cc:347] Connected to a master server at 127.2.39.254:45271
I20260926 06:25:21.831037 6103 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:21.831220 6103 heartbeater.cc:511] Master 127.2.39.254:45271 requested a full tablet report, sending...
I20260926 06:25:21.831559 5778 ts_manager.cc:194] Registered new tserver with Master: 9ac1c6ca61824c9ba18ea74f70f516c5 (127.2.39.194:44283)
I20260926 06:25:21.831920 5778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:50055
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:21.854465 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45271
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:21.934759 6108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:21.934947 6108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:21.934970 6108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:21.936033 6108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:21.936069 6108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:21.936093 6108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:21.937470 6108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:45271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:21.937711 6108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:21.937883 6108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:21.942701 6113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:21.942749 6116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:21.942756 6108 server_base.cc:1034] running on GCE node
W20260926 06:25:21.942721 6114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:21.943148 6108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:21.943374 6108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:21.944528 6108 hybrid_clock.cc:648] HybridClock initialized: now 1790403921944525 us; error 35 us; skew 500 ppm
Sep 26 06:25:21 dist-test-slave-fkbn krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:21.946919 6108 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:21.947358 6108 webserver.cc:460] Webserver started at http://127.2.39.195:39089/ using document root <none> and password file <none>
I20260926 06:25:21.947606 6108 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:21.947677 6108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:21.947822 6108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:21.948853 6108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "19811aea862042bea378f30b7aeafcc2"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "04c0b7745d2194d84ffc734602693c08"
server_key_iv: "9fda4203e5150f58200785eed3bf8acf"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.949194 6108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "19811aea862042bea378f30b7aeafcc2"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "04c0b7745d2194d84ffc734602693c08"
server_key_iv: "9fda4203e5150f58200785eed3bf8acf"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.950603 6108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:21.951313 6123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.951483 6108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.951575 6108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "19811aea862042bea378f30b7aeafcc2"
format_stamp: "Formatted at 2026-09-26 06:25:21 on dist-test-slave-fkbn"
server_key: "04c0b7745d2194d84ffc734602693c08"
server_key_iv: "9fda4203e5150f58200785eed3bf8acf"
server_key_version: "encryptionkey@0"
I20260926 06:25:21.951651 6108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:21.976694 6108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:21.978950 6108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:21.979106 6108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:21.979334 6108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:21.979640 6108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:21.979693 6108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.979736 6108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:21.979766 6108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:21.985994 6108 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:41635
I20260926 06:25:21.986126 6236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:41635 every 8 connection(s)
I20260926 06:25:21.986390 6108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:21.988888 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 6108
I20260926 06:25:21.988996 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:21.989184 2207 external_mini_cluster.cc:1428] Setting key 2eea9d5e770bbef265d6596c28431622
Sep 26 06:25:21 dist-test-slave-fkbn krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403921, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:21.996088 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:21.987172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:33055 (local address 127.2.39.254:45271)
0926 06:25:21.987251 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:21.987253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:21.987374 (+ 121us) server_negotiation.cc:407] Connection header received
0926 06:25:21.988284 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:21.988285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:21.988294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:21.988346 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:21.988917 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.989724 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.990763 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.990948 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.991006 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:21.993273 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:21.993274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:21.993275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:21.993282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:21.994723 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:21.995155 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:21.995156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:21.995157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:21.995217 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:21.995458 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:21.995459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:21.995459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:21.995582 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:21.995653 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:21.995921 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 06:25:21.995992 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 06:25:21.996297 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:21.987108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:45271 (local address 127.2.39.195:33055)
0926 06:25:21.987262 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:21.987308 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:21.988198 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:21.988401 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:21.988404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:21.988646 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:21.988833 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:21.988837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.989816 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.989817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:21.990691 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:21.990693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:21.991037 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:21.991038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:21.991139 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:21.991697 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:21.991704 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:21.993178 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:21.994823 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:21.994829 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:21.994831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:21.995053 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:21.995282 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:21.995283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:21.995283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:21.995392 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:21.995658 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:21.995664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:21.995797 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 06:25:21.995901 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 06:25:21.996066 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260926 06:25:21.996731 6237 heartbeater.cc:347] Connected to a master server at 127.2.39.254:45271
I20260926 06:25:21.996829 6237 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:21.997028 6237 heartbeater.cc:511] Master 127.2.39.254:45271 requested a full tablet report, sending...
I20260926 06:25:21.997371 5778 ts_manager.cc:194] Registered new tserver with Master: 19811aea862042bea378f30b7aeafcc2 (127.2.39.195:41635)
I20260926 06:25:21.997771 5778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:33055
I20260926 06:25:22.000689 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:25:22 dist-test-slave-fkbn krb5kdc[5733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:25:22 dist-test-slave-fkbn krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:25:22.024464 6244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:22.016269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.2.39.193:45903)
0926 06:25:22.016449 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:22.016454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:22.016465 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:25:22.016490 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:22.016492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:22.016505 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:22.016574 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:22.017135 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:22.017899 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:22.018597 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:22.018909 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:22.018963 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:22.021270 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:22.021277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:22.021277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:22.021283 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:22.022791 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:22.023219 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:22.023223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:22.023224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:22.023438 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:22.023804 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:22.023805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:22.023806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:22.024002 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:22.024136 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:22.024209 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 06:25:22.024272 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":100,"threads_started":1}
Sep 26 06:25:22 dist-test-slave-fkbn krb5kdc[5733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:22.032147 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:22.025810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.2.39.254:45271)
0926 06:25:22.025889 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:22.025891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:22.025903 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:25:22.025931 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:22.025931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:22.025939 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:22.025993 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:22.026413 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:22.026911 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:22.027527 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:22.027708 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:22.027759 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:22.029910 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:22.029911 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:22.029911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:22.029917 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:22.031154 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:22.031531 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:22.031531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:22.031532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:22.031577 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:22.031740 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:22.031740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:22.031740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:22.031861 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:22.031931 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:22.032015 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 06:25:22.032086 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 06:25:22.032394 5778 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47822
I20260926 06:25:22.032728 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 5840
I20260926 06:25:22.037016 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 5974
I20260926 06:25:22.041255 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 6108
I20260926 06:25:22.045287 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 5746
2026-09-26T06:25:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1852 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:22 dist-test-slave-fkbn krb5kdc[6249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:22 dist-test-slave-fkbn krb5kdc[6249](info): set up 2 sockets
Sep 26 06:25:22 dist-test-slave-fkbn krb5kdc[6249](info): commencing operation
krb5kdc: starting...
W20260926 06:25:23.073709 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:23 dist-test-slave-fkbn krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:23.197373 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36997
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:36997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:23.283548 6262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:23.283774 6262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:23.283811 6262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:23.284945 6262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:23.285009 6262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:23.285034 6262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:23.285063 6262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:23.285092 6262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:23.286628 6262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:36997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:23.286914 6262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:23.287141 6262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:23.292075 6270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:23.292145 6267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:23.292066 6268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:23.292272 6262 server_base.cc:1034] running on GCE node
I20260926 06:25:23.292613 6262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:23.292991 6262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:23.294148 6262 hybrid_clock.cc:648] HybridClock initialized: now 1790403923294121 us; error 37 us; skew 500 ppm
Sep 26 06:25:23 dist-test-slave-fkbn krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403923, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:23.297114 6262 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:23.297621 6262 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:23.297677 6262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:23.297811 6262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:23.298878 6262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "e501765f98cc4ad384e87d8c80adbadb"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "6818e4570635f4d7497c7397b58730bf"
server_key_iv: "0e8f9c9e2fdd63b2dc8607e662a50e5d"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.299228 6262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "e501765f98cc4ad384e87d8c80adbadb"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "6818e4570635f4d7497c7397b58730bf"
server_key_iv: "0e8f9c9e2fdd63b2dc8607e662a50e5d"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.300534 6262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:23.301339 6276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:23.301560 6262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:23.301707 6262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "e501765f98cc4ad384e87d8c80adbadb"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "6818e4570635f4d7497c7397b58730bf"
server_key_iv: "0e8f9c9e2fdd63b2dc8607e662a50e5d"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.301789 6262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:23.323803 6262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:23.325789 6262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:23.325959 6262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:23.330159 6262 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:36997
I20260926 06:25:23.330183 6328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:36997 every 8 connection(s)
I20260926 06:25:23.330488 6262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:23.331408 6329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:23.332546 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 6262
I20260926 06:25:23.332666 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:23.332834 2207 external_mini_cluster.cc:1428] Setting key 4232ce7d2c1fdefd635659bd9fad1a95
I20260926 06:25:23.334304 6329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb: Bootstrap starting.
I20260926 06:25:23.335057 6329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:23.335368 6329 log.cc:824] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:23.336074 6329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb: No bootstrap required, opened a new log
I20260926 06:25:23.338474 6329 raft_consensus.cc:348] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501765f98cc4ad384e87d8c80adbadb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36997 } }
I20260926 06:25:23.338652 6329 raft_consensus.cc:374] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:23.338689 6329 raft_consensus.cc:729] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e501765f98cc4ad384e87d8c80adbadb, State: Initialized, Role: FOLLOWER
I20260926 06:25:23.338827 6329 consensus_queue.cc:260] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501765f98cc4ad384e87d8c80adbadb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36997 } }
I20260926 06:25:23.338915 6329 raft_consensus.cc:388] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:23.338953 6329 raft_consensus.cc:482] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:23.339007 6329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:23.339751 6329 raft_consensus.cc:504] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501765f98cc4ad384e87d8c80adbadb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36997 } }
I20260926 06:25:23.339840 6329 leader_election.cc:302] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e501765f98cc4ad384e87d8c80adbadb; no voters:
I20260926 06:25:23.339988 6329 leader_election.cc:288] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:23.340041 6334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [term 1 FOLLOWER]: Leader election won for term 1
Sep 26 06:25:23 dist-test-slave-fkbn krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:23.340197 6334 raft_consensus.cc:686] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [term 1 LEADER]: Becoming Leader. State: Replica: e501765f98cc4ad384e87d8c80adbadb, State: Running, Role: LEADER
I20260926 06:25:23.340262 6329 sys_catalog.cc:564] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:23.340292 6334 consensus_queue.cc:237] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501765f98cc4ad384e87d8c80adbadb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36997 } }
I20260926 06:25:23.340595 6334 sys_catalog.cc:455] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e501765f98cc4ad384e87d8c80adbadb. Latest consensus state: current_term: 1 leader_uuid: "e501765f98cc4ad384e87d8c80adbadb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501765f98cc4ad384e87d8c80adbadb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36997 } } }
I20260926 06:25:23.340605 6335 sys_catalog.cc:455] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e501765f98cc4ad384e87d8c80adbadb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e501765f98cc4ad384e87d8c80adbadb" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36997 } } }
I20260926 06:25:23.340677 6334 sys_catalog.cc:458] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:23.340705 6335 sys_catalog.cc:458] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:23.341035 6341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:23.341526 6341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:23.343379 6341 catalog_manager.cc:1347] Generated new cluster ID: 06e118a36a194c5cb318553023993ba3
I20260926 06:25:23.343456 6341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:23.344962 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.333371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.2.39.254:36997)
0926 06:25:23.333542 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.333550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.333592 (+ 42us) server_negotiation.cc:407] Connection header received
0926 06:25:23.334069 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.334076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.334122 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.334243 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:23.335028 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.336246 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.337277 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.337805 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.337898 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.340872 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:23.340877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:23.340877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:23.340889 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:23.342952 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:23.343365 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:23.343368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:23.343368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:23.343482 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:23.343764 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:23.343765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:23.343766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:23.344052 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:23.344185 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.344610 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.344741 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260926 06:25:23.348886 6341 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:23.349378 6341 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:23.352895 6341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb: Generated new TSK 0
I20260926 06:25:23.353052 6341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:23.384846 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36997
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:23.465951 6355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:23.466148 6355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:23.466171 6355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:23.467317 6355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:23.467365 6355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:23.467419 6355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:23.468894 6355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:23.469074 6355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:23.469313 6355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:23.474236 6361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:23.474263 6360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:23.474256 6363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:23.474512 6355 server_base.cc:1034] running on GCE node
I20260926 06:25:23.474715 6355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:23.474937 6355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:23.476130 6355 hybrid_clock.cc:648] HybridClock initialized: now 1790403923476114 us; error 24 us; skew 500 ppm
Sep 26 06:25:23 dist-test-slave-fkbn krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403923, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:23.478650 6355 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:23.478991 6355 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:23.479038 6355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:23.479123 6355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:23.480193 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "2d5796d6ee164e059a4e53344226804c"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "5f7805f053522dbd21b4c29d38144027"
server_key_iv: "fb2369c2f24aeeb370999138398d0122"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.480486 6355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "2d5796d6ee164e059a4e53344226804c"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "5f7805f053522dbd21b4c29d38144027"
server_key_iv: "fb2369c2f24aeeb370999138398d0122"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.481786 6355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:23.482540 6369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:23.482728 6355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:23.482813 6355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "2d5796d6ee164e059a4e53344226804c"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "5f7805f053522dbd21b4c29d38144027"
server_key_iv: "fb2369c2f24aeeb370999138398d0122"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.482882 6355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:23.492506 6355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:23.494426 6355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:23.494526 6355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:23.494695 6355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:23.494942 6355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:23.494972 6355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:23.494992 6355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:23.495004 6355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:23.500694 6355 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:40041
I20260926 06:25:23.500736 6482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:40041 every 8 connection(s)
I20260926 06:25:23.500988 6355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:25:23 dist-test-slave-fkbn krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403923, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:23.509500 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.501673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:41415 (local address 127.2.39.254:36997)
0926 06:25:23.501714 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.501716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.501879 (+ 163us) server_negotiation.cc:407] Connection header received
0926 06:25:23.502414 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.502415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.502425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.502481 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:23.503448 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.504020 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.505014 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.505210 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.505265 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.507324 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:23.507326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:23.507326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:23.507333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:23.508451 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:23.508783 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:23.508784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:23.508784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:23.508831 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:23.509024 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:23.509025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:23.509025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:23.509116 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:23.509164 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.509378 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.509438 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:25:23.509744 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.501651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36997 (local address 127.2.39.193:41415)
0926 06:25:23.501783 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:23.501815 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:23.502258 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:23.502563 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:23.502567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:23.502806 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:23.503293 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.503297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.504101 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.504101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.504943 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.504945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.505282 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.505283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.505400 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.505942 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:23.505950 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:23.507245 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:23.508522 (+ 1277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:23.508530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:23.508531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:23.508724 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:23.508885 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:23.508885 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:23.508886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:23.508973 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:23.509170 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:23.509174 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:23.509277 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 06:25:23.509356 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:25:23.509463 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260926 06:25:23.509771 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 6355
I20260926 06:25:23.509868 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:23.510098 2207 external_mini_cluster.cc:1428] Setting key 75522fda797807970b9ee8b7123e6a0d
I20260926 06:25:23.510310 6483 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36997
I20260926 06:25:23.510412 6483 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:23.510555 6483 heartbeater.cc:511] Master 127.2.39.254:36997 requested a full tablet report, sending...
I20260926 06:25:23.511080 6293 ts_manager.cc:194] Registered new tserver with Master: 2d5796d6ee164e059a4e53344226804c (127.2.39.193:40041)
I20260926 06:25:23.511674 6293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:41415
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:23.537094 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36997
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:23.620508 6488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:23.620728 6488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:23.620765 6488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:23.621942 6488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:23.622000 6488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:23.622045 6488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:23.623500 6488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:23.623754 6488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:23.623977 6488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:23.628850 6494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:23.628892 6493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:23.628928 6488 server_base.cc:1034] running on GCE node
W20260926 06:25:23.628868 6496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:23.629269 6488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:23.629474 6488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:23.630679 6488 hybrid_clock.cc:648] HybridClock initialized: now 1790403923630660 us; error 28 us; skew 500 ppm
Sep 26 06:25:23 dist-test-slave-fkbn krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403923, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:23.633291 6488 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:23.633718 6488 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:23.633771 6488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:23.633867 6488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:23.634938 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "ed912c9890193d7621026ffda8b640cc"
server_key_iv: "cb1dba5e3b36c578d616e1a11b87d71b"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.635267 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "ed912c9890193d7621026ffda8b640cc"
server_key_iv: "cb1dba5e3b36c578d616e1a11b87d71b"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.636567 6488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:23.637288 6502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:23.637501 6488 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:23.637615 6488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "ed912c9890193d7621026ffda8b640cc"
server_key_iv: "cb1dba5e3b36c578d616e1a11b87d71b"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.637692 6488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:23.657516 6488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:23.659807 6488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:23.659912 6488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:23.660091 6488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:23.660404 6488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:23.660436 6488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:23.660458 6488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:23.660508 6488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:23.666204 6488 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:41601
I20260926 06:25:23.666271 6615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:41601 every 8 connection(s)
I20260926 06:25:23.666530 6488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:23.672467 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 6488
I20260926 06:25:23.672564 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:23.672783 2207 external_mini_cluster.cc:1428] Setting key c7bb06b2ba33175c0b2845d7829c6ae6
Sep 26 06:25:23 dist-test-slave-fkbn krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403923, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:23.676168 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.667268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:41939 (local address 127.2.39.254:36997)
0926 06:25:23.667402 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.667405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.667559 (+ 154us) server_negotiation.cc:407] Connection header received
0926 06:25:23.668122 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.668123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.668137 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.668207 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:23.669140 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.669740 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.670788 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.671035 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.671112 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.673400 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:23.673402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:23.673402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:23.673409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:23.674804 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:23.675236 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:23.675237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:23.675238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:23.675297 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:23.675555 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:23.675556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:23.675556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:23.675665 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:23.675733 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.676004 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.676078 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:25:23.676403 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.667216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36997 (local address 127.2.39.194:41939)
0926 06:25:23.667406 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:23.667451 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:23.668005 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:23.668276 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:23.668280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:23.668542 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:23.669028 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.669033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.669863 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.669864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.670677 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.670679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.671133 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.671134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.671293 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.671783 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:23.671791 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:23.673304 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:23.674906 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:23.674912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:23.674914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:23.675146 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:23.675362 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:23.675363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:23.675363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:23.675475 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:23.675735 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:23.675742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:23.675873 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 06:25:23.675971 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 06:25:23.676131 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260926 06:25:23.676891 6616 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36997
I20260926 06:25:23.676970 6616 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:23.677125 6616 heartbeater.cc:511] Master 127.2.39.254:36997 requested a full tablet report, sending...
I20260926 06:25:23.677534 6293 ts_manager.cc:194] Registered new tserver with Master: af9dcd6bf7b546f5b3e5fd0af8c3f2a4 (127.2.39.194:41601)
I20260926 06:25:23.677923 6293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:41939
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:23.698580 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36997
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:23.779315 6621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:23.779527 6621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:23.779551 6621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:23.780673 6621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:23.780714 6621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:23.780740 6621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:23.782217 6621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:23.782466 6621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:23.782759 6621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:23.787879 6629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:23.787910 6626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:23.787926 6627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:23.788108 6621 server_base.cc:1034] running on GCE node
I20260926 06:25:23.788347 6621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:23.788592 6621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:23.789767 6621 hybrid_clock.cc:648] HybridClock initialized: now 1790403923789750 us; error 30 us; skew 500 ppm
Sep 26 06:25:23 dist-test-slave-fkbn krb5kdc[6249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403923, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:23.792302 6621 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:23.792709 6621 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:23.792778 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:23.792914 6621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:23.794013 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "b048f032d3c4411cab1e49112e4c7cf6"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "6aa786ee2f795c4b81bc4de9f85d8bdb"
server_key_iv: "e33a0de88d01aa66ceaa371e674b72ee"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.794364 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "b048f032d3c4411cab1e49112e4c7cf6"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "6aa786ee2f795c4b81bc4de9f85d8bdb"
server_key_iv: "e33a0de88d01aa66ceaa371e674b72ee"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.795675 6621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:23.796422 6635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:23.796630 6621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:23.796753 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "b048f032d3c4411cab1e49112e4c7cf6"
format_stamp: "Formatted at 2026-09-26 06:25:23 on dist-test-slave-fkbn"
server_key: "6aa786ee2f795c4b81bc4de9f85d8bdb"
server_key_iv: "e33a0de88d01aa66ceaa371e674b72ee"
server_key_version: "encryptionkey@0"
I20260926 06:25:23.796838 6621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:23.823076 6621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:23.825644 6621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:23.825807 6621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:23.826062 6621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:23.826396 6621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:23.826452 6621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:23.826493 6621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:23.826536 6621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:23.832245 6621 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:37745
I20260926 06:25:23.832278 6748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:37745 every 8 connection(s)
I20260926 06:25:23.832559 6621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:23.833192 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 6621
I20260926 06:25:23.833325 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:23.833519 2207 external_mini_cluster.cc:1428] Setting key 408dacc405537661ab9667c3d277a1f1
Sep 26 06:25:23 dist-test-slave-fkbn krb5kdc[6249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403923, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:23.842682 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.833333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:52137 (local address 127.2.39.254:36997)
0926 06:25:23.833391 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.833393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.833614 (+ 221us) server_negotiation.cc:407] Connection header received
0926 06:25:23.834535 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.834536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.834548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.834590 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:23.835580 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.836090 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.837105 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.837313 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.837402 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.839621 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:23.839622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:23.839622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:23.839629 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:23.841137 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:23.841638 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:23.841639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:23.841639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:23.841699 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:23.842080 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:23.842081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:23.842081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:23.842209 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:23.842280 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.842544 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.842615 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:25:23.842831 6751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.833289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36997 (local address 127.2.39.195:52137)
0926 06:25:23.833453 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:23.833496 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:23.834347 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:23.834677 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:23.834680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:23.834958 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:23.835480 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.835485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.836189 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.836190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.837008 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.837010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.837412 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.837413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.837540 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.838139 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:23.838148 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:23.839536 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:23.841205 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:23.841211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:23.841213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:23.841496 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:23.841787 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:23.841788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:23.841789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:23.841987 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:23.842278 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:23.842282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:23.842399 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 06:25:23.842492 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 06:25:23.842615 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260926 06:25:23.843372 6749 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36997
I20260926 06:25:23.843484 6749 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:23.843605 6749 heartbeater.cc:511] Master 127.2.39.254:36997 requested a full tablet report, sending...
I20260926 06:25:23.843974 6293 ts_manager.cc:194] Registered new tserver with Master: b048f032d3c4411cab1e49112e4c7cf6 (127.2.39.195:37745)
I20260926 06:25:23.844296 6293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:52137
I20260926 06:25:23.845247 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:25:23.851948 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.846541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.2.39.254:36997)
0926 06:25:23.846577 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.846578 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.846692 (+ 114us) server_negotiation.cc:407] Connection header received
0926 06:25:23.846728 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.846728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.846736 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.846783 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:23.847281 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.847729 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.848387 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.848583 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.848640 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.849777 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:23.849778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:23.849778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:23.849783 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:23.850868 (+ 1085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:23.851229 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:23.851230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:23.851231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:23.851285 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:23.851479 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:23.851480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:23.851481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:23.851634 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:23.851732 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.851814 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.851886 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:25:23.853682 6293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:23.855213 6293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:23.861438 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.857649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.2.39.195:37745)
0926 06:25:23.857832 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.857837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.857849 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:25:23.857888 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.857890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.857903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.857999 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:23.858303 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.859411 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.860546 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.861054 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.861142 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.861201 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.861318 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.861354 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260926 06:25:23.861426 6760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.857408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:37745 (local address 127.0.0.1:53110)
0926 06:25:23.857779 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:23.857789 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:23.857827 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:23.858009 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:23.858012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:23.858015 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:23.858092 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.858096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.859510 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.859511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.860464 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.860466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.861141 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.861142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.861215 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.861229 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:25:23.861301 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 06:25:23.861354 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":37,"thread_start_us":88,"threads_started":1}
I20260926 06:25:23.861807 6761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.857370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:40041 (local address 127.0.0.1:44630)
0926 06:25:23.857537 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:23.857612 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:23.857678 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:23.858126 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:23.858127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:23.858128 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:23.858167 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.858168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.859151 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.859152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.860493 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.860495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.861521 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.861522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.861615 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.861619 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:23.861676 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 06:25:23.861722 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260926 06:25:23.861918 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.857658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.2.39.194:41601)
0926 06:25:23.857878 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.857882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.857894 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:25:23.857938 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.857940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.857957 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.858040 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:23.858275 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.859389 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.860695 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.861378 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.861440 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.861487 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.861768 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.861809 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260926 06:25:23.861987 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.857370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:41601 (local address 127.0.0.1:40270)
0926 06:25:23.857756 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:23.857767 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:23.857840 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:23.858135 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:23.858137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:23.858139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:23.858196 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.858198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.859498 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.859499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.860602 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.860604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.861520 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.861521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.861615 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.861621 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:25:23.861676 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:25:23.861722 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":34,"thread_start_us":52,"threads_started":1}
I20260926 06:25:23.862298 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.857564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44630 (local address 127.2.39.193:40041)
0926 06:25:23.857966 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.857969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.857980 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:25:23.858021 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.858023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.858038 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.858191 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:23.858275 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.859039 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.860599 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.861315 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.861417 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.861479 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.862166 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.862209 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":75,"threads_started":1}
I20260926 06:25:23.862283 6683 tablet_service.cc:1467] Processing CreateTablet for tablet 38d93f8ae47b41bc99e9e08501d50b09 (DEFAULT_TABLE table=test-table [id=f7fb5f085ec644478d6a209be6f7b561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:23.862689 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d93f8ae47b41bc99e9e08501d50b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:23.862638 6550 tablet_service.cc:1467] Processing CreateTablet for tablet 38d93f8ae47b41bc99e9e08501d50b09 (DEFAULT_TABLE table=test-table [id=f7fb5f085ec644478d6a209be6f7b561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:23.862900 6550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d93f8ae47b41bc99e9e08501d50b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:23.862908 6417 tablet_service.cc:1467] Processing CreateTablet for tablet 38d93f8ae47b41bc99e9e08501d50b09 (DEFAULT_TABLE table=test-table [id=f7fb5f085ec644478d6a209be6f7b561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:23.863209 6417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d93f8ae47b41bc99e9e08501d50b09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:23.865490 6766 tablet_bootstrap.cc:492] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: Bootstrap starting.
I20260926 06:25:23.866115 6766 tablet_bootstrap.cc:654] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:23.866333 6767 tablet_bootstrap.cc:492] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c: Bootstrap starting.
I20260926 06:25:23.866770 6766 log.cc:824] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:23.867022 6767 tablet_bootstrap.cc:654] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:23.867662 6766 tablet_bootstrap.cc:492] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: No bootstrap required, opened a new log
I20260926 06:25:23.867657 6767 log.cc:824] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:23.867735 6766 ts_tablet_manager.cc:1397] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:25:23.868012 6768 tablet_bootstrap.cc:492] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: Bootstrap starting.
I20260926 06:25:23.868356 6767 tablet_bootstrap.cc:492] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c: No bootstrap required, opened a new log
I20260926 06:25:23.868577 6767 ts_tablet_manager.cc:1397] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:25:23.868629 6768 tablet_bootstrap.cc:654] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:23.869222 6768 log.cc:824] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:23.870018 6768 tablet_bootstrap.cc:492] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: No bootstrap required, opened a new log
I20260926 06:25:23.870095 6768 ts_tablet_manager.cc:1397] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:25:23.870119 6766 raft_consensus.cc:348] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41601 } } peers { permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 } } peers { permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 } }
I20260926 06:25:23.870311 6766 raft_consensus.cc:374] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:23.870373 6766 raft_consensus.cc:729] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9dcd6bf7b546f5b3e5fd0af8c3f2a4, State: Initialized, Role: FOLLOWER
I20260926 06:25:23.870347 6767 raft_consensus.cc:348] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41601 } } peers { permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 } } peers { permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 } }
I20260926 06:25:23.870509 6767 raft_consensus.cc:374] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:23.870493 6766 consensus_queue.cc:260] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41601 } } peers { permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 } } peers { permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 } }
I20260926 06:25:23.870563 6767 raft_consensus.cc:729] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5796d6ee164e059a4e53344226804c, State: Initialized, Role: FOLLOWER
I20260926 06:25:23.870695 6767 consensus_queue.cc:260] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41601 } } peers { permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 } } peers { permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 } }
I20260926 06:25:23.870776 6616 heartbeater.cc:503] Master 127.2.39.254:36997 was elected leader, sending a full tablet report...
I20260926 06:25:23.870779 6766 ts_tablet_manager.cc:1428] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:25:23.871038 6767 ts_tablet_manager.cc:1428] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:25:23.871161 6483 heartbeater.cc:503] Master 127.2.39.254:36997 was elected leader, sending a full tablet report...
I20260926 06:25:23.872299 6768 raft_consensus.cc:348] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41601 } } peers { permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 } } peers { permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 } }
I20260926 06:25:23.872423 6768 raft_consensus.cc:374] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:23.872473 6768 raft_consensus.cc:729] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b048f032d3c4411cab1e49112e4c7cf6, State: Initialized, Role: FOLLOWER
I20260926 06:25:23.872594 6768 consensus_queue.cc:260] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41601 } } peers { permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 } } peers { permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 } }
I20260926 06:25:23.872864 6768 ts_tablet_manager.cc:1428] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:25:23.872869 6749 heartbeater.cc:503] Master 127.2.39.254:36997 was elected leader, sending a full tablet report...
I20260926 06:25:23.890414 6772 raft_consensus.cc:482] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:23.890517 6772 raft_consensus.cc:504] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41601 } } peers { permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 } } peers { permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 } }
I20260926 06:25:23.890836 6772 leader_election.cc:288] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d5796d6ee164e059a4e53344226804c (127.2.39.193:40041), b048f032d3c4411cab1e49112e4c7cf6 (127.2.39.195:37745)
I20260926 06:25:23.894914 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.891082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:45119 (local address 127.2.39.193:40041)
0926 06:25:23.891142 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.891145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.891158 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:25:23.891191 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.891192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.891207 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.891280 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:23.891465 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.892309 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.893914 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.894544 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.894631 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.894640 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.894802 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.894840 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 06:25:23.894917 6764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.891170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:46161 (local address 127.2.39.195:37745)
0926 06:25:23.891211 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.891213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.891355 (+ 142us) server_negotiation.cc:407] Connection header received
0926 06:25:23.891389 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.891390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.891400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.891497 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:23.891629 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.892395 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.893914 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.894544 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.894631 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.894640 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.894800 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.894840 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:25:23.894915 6618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.891014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:40041 (local address 127.2.39.194:45119)
0926 06:25:23.891076 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:23.891088 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:23.891136 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:23.891294 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:23.891295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:23.891297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:23.891374 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.891376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.892484 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.892484 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.893814 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.893817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.894634 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.894635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.894719 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.894721 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:23.894776 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:25:23.894827 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":77}
I20260926 06:25:23.894922 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.891079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:37745 (local address 127.2.39.194:46161)
0926 06:25:23.891271 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:23.891280 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:23.891329 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:23.891505 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:23.891505 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:23.891506 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:23.891557 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.891558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.892476 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.892477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.893814 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:23.893817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.894634 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.894635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:23.894713 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.894717 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:23.894776 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:25:23.894827 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":48,"thread_start_us":123,"threads_started":1}
I20260926 06:25:23.895346 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d93f8ae47b41bc99e9e08501d50b09" candidate_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b048f032d3c4411cab1e49112e4c7cf6" is_pre_election: true
I20260926 06:25:23.895383 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d93f8ae47b41bc99e9e08501d50b09" candidate_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5796d6ee164e059a4e53344226804c" is_pre_election: true
I20260926 06:25:23.895514 6703 raft_consensus.cc:2454] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af9dcd6bf7b546f5b3e5fd0af8c3f2a4 in term 0.
I20260926 06:25:23.895579 6437 raft_consensus.cc:2454] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af9dcd6bf7b546f5b3e5fd0af8c3f2a4 in term 0.
I20260926 06:25:23.895756 6505 leader_election.cc:302] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af9dcd6bf7b546f5b3e5fd0af8c3f2a4, b048f032d3c4411cab1e49112e4c7cf6; no voters:
I20260926 06:25:23.895907 6772 raft_consensus.cc:2781] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:23.895977 6772 raft_consensus.cc:482] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:23.896006 6772 raft_consensus.cc:3037] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:23.896781 6772 raft_consensus.cc:504] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41601 } } peers { permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 } } peers { permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 } }
I20260926 06:25:23.896921 6772 leader_election.cc:288] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [CANDIDATE]: Term 1 election: Requested vote from peers 2d5796d6ee164e059a4e53344226804c (127.2.39.193:40041), b048f032d3c4411cab1e49112e4c7cf6 (127.2.39.195:37745)
I20260926 06:25:23.897070 6437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d93f8ae47b41bc99e9e08501d50b09" candidate_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d5796d6ee164e059a4e53344226804c"
I20260926 06:25:23.897105 6703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d93f8ae47b41bc99e9e08501d50b09" candidate_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b048f032d3c4411cab1e49112e4c7cf6"
I20260926 06:25:23.897177 6703 raft_consensus.cc:3037] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:23.897192 6437 raft_consensus.cc:3037] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:23.897957 6703 raft_consensus.cc:2454] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af9dcd6bf7b546f5b3e5fd0af8c3f2a4 in term 1.
I20260926 06:25:23.898114 6437 raft_consensus.cc:2454] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af9dcd6bf7b546f5b3e5fd0af8c3f2a4 in term 1.
I20260926 06:25:23.898125 6505 leader_election.cc:302] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af9dcd6bf7b546f5b3e5fd0af8c3f2a4, b048f032d3c4411cab1e49112e4c7cf6; no voters:
I20260926 06:25:23.898226 6772 raft_consensus.cc:2781] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:23.898391 6772 raft_consensus.cc:686] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 1 LEADER]: Becoming Leader. State: Replica: af9dcd6bf7b546f5b3e5fd0af8c3f2a4, State: Running, Role: LEADER
I20260926 06:25:23.898494 6772 consensus_queue.cc:237] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41601 } } peers { permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 } } peers { permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 } }
I20260926 06:25:23.899289 6293 catalog_manager.cc:5495] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to af9dcd6bf7b546f5b3e5fd0af8c3f2a4 (127.2.39.194). New cstate: current_term: 1 leader_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af9dcd6bf7b546f5b3e5fd0af8c3f2a4" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:23.910171 6765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.907565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.2.39.194:41601)
0926 06:25:23.907615 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:23.907616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:23.907626 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:23.907669 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:23.907669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:23.907687 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:23.907765 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:23.907926 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.908529 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.909246 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:23.909489 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:23.909546 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:23.909839 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:23.909947 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:23.910034 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:23.910071 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 06:25:23.910095 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:25:23.912015 6703 raft_consensus.cc:1261] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6 [term 1 FOLLOWER]: Refusing update from remote peer af9dcd6bf7b546f5b3e5fd0af8c3f2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:23.912016 6437 raft_consensus.cc:1261] T 38d93f8ae47b41bc99e9e08501d50b09 P 2d5796d6ee164e059a4e53344226804c [term 1 FOLLOWER]: Refusing update from remote peer af9dcd6bf7b546f5b3e5fd0af8c3f2a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:23.912365 6772 consensus_queue.cc:1035] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b048f032d3c4411cab1e49112e4c7cf6" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:23.912549 6776 consensus_queue.cc:1035] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d5796d6ee164e059a4e53344226804c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:23.914738 6779 mvcc.cc:205] Tried to move back new op lower bound from 7333494472340766720 to 7333494472288911360. Current Snapshot: MvccSnapshot[applied={T|T < 7333494472340766720}]
I20260926 06:25:23.915431 6780 mvcc.cc:205] Tried to move back new op lower bound from 7333494472340766720 to 7333494472288911360. Current Snapshot: MvccSnapshot[applied={T|T < 7333494472340766720}]
I20260926 06:25:23.917481 6292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:23.917615 6292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:23.918357 6292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e501765f98cc4ad384e87d8c80adbadb: Sending DeleteTablet for 3 replicas of tablet 38d93f8ae47b41bc99e9e08501d50b09
I20260926 06:25:23.918692 6550 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d93f8ae47b41bc99e9e08501d50b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:23 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:40270
I20260926 06:25:23.918707 6417 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d93f8ae47b41bc99e9e08501d50b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:23 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:44630
I20260926 06:25:23.919005 6789 tablet_replica.cc:330] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: stopping tablet replica
I20260926 06:25:23.919142 6789 raft_consensus.cc:2229] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:23.919152 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 6355
I20260926 06:25:23.919132 6683 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d93f8ae47b41bc99e9e08501d50b09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:23 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:53110
I20260926 06:25:23.920034 6789 raft_consensus.cc:2258] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:23.920485 6789 ts_tablet_manager.cc:1905] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:23.922003 6789 ts_tablet_manager.cc:1918] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:23.922111 6789 log.cc:1196] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/38d93f8ae47b41bc99e9e08501d50b09
I20260926 06:25:23.922396 6789 ts_tablet_manager.cc:1939] T 38d93f8ae47b41bc99e9e08501d50b09 P af9dcd6bf7b546f5b3e5fd0af8c3f2a4: Deleting consensus metadata
I20260926 06:25:23.922765 6278 catalog_manager.cc:4841] TS af9dcd6bf7b546f5b3e5fd0af8c3f2a4 (127.2.39.194:41601): tablet 38d93f8ae47b41bc99e9e08501d50b09 (table test-table [id=f7fb5f085ec644478d6a209be6f7b561]) successfully deleted
W20260926 06:25:23.925132 6277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:40041 (error 108)
I20260926 06:25:23.925366 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 6488
I20260926 06:25:23.926002 6762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:23.925551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:40041 (local address 127.0.0.1:44644)
0926 06:25:23.925920 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:23.925962 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:40041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348}
I20260926 06:25:23.926002 6791 tablet_replica.cc:330] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: stopping tablet replica
I20260926 06:25:23.926134 6791 raft_consensus.cc:2229] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 06:25:23.926137 6277 catalog_manager.cc:4584] TS 2d5796d6ee164e059a4e53344226804c (127.2.39.193:40041): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38d93f8ae47b41bc99e9e08501d50b09: Network error: Client connection negotiation failed: client connection to 127.2.39.193:40041: connect: Connection refused (error 111)
I20260926 06:25:23.926251 6791 raft_consensus.cc:2258] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:23.926712 6791 ts_tablet_manager.cc:1905] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:23.928150 6791 ts_tablet_manager.cc:1918] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:23.928207 6791 log.cc:1196] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/38d93f8ae47b41bc99e9e08501d50b09
I20260926 06:25:23.928463 6791 ts_tablet_manager.cc:1939] T 38d93f8ae47b41bc99e9e08501d50b09 P b048f032d3c4411cab1e49112e4c7cf6: Deleting consensus metadata
I20260926 06:25:23.928874 6279 catalog_manager.cc:4841] TS b048f032d3c4411cab1e49112e4c7cf6 (127.2.39.195:37745): tablet 38d93f8ae47b41bc99e9e08501d50b09 (table test-table [id=f7fb5f085ec644478d6a209be6f7b561]) successfully deleted
I20260926 06:25:23.931161 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 6621
I20260926 06:25:23.935817 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 6262
2026-09-26T06:25:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1891 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T06:25:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:23Z Disabled control of system clock
I20260926 06:25:23.963315 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38415
--webserver_interface=127.2.39.254
--webserver_port=0
--builtin_ntp_servers=127.2.39.212:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:38415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 06:25:24.045523 6796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:24.045892 6796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:24.045917 6796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:24.047179 6796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:24.047228 6796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:24.047243 6796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:24.047257 6796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:24.047305 6796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:24.048681 6796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:38415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:24.048964 6796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:24.049285 6796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:25:24.054313 6796 server_base.cc:1034] running on GCE node
W20260926 06:25:24.054263 6802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:24.054263 6801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:24.054263 6804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:24.054750 6796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:24.055097 6796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:24.056288 6796 hybrid_clock.cc:648] HybridClock initialized: now 1790403924056259 us; error 58 us; skew 500 ppm
I20260926 06:25:24.057791 6796 webserver.cc:460] Webserver started at http://127.2.39.254:39837/ using document root <none> and password file <none>
I20260926 06:25:24.058050 6796 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:24.058120 6796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:24.058270 6796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:24.059556 6796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "86e94bbad7d945a484217584d2d9b347"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "049eb5f70d932b02d27c7f195e27598f"
server_key_iv: "6a7104a8c74ca1e21f4f083b6daa3d24"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.059947 6796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "86e94bbad7d945a484217584d2d9b347"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "049eb5f70d932b02d27c7f195e27598f"
server_key_iv: "6a7104a8c74ca1e21f4f083b6daa3d24"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.061270 6796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:24.062100 6810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.062276 6796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.062381 6796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "86e94bbad7d945a484217584d2d9b347"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "049eb5f70d932b02d27c7f195e27598f"
server_key_iv: "6a7104a8c74ca1e21f4f083b6daa3d24"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.062462 6796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:24.065840 6796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:24.066119 6796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:24.066267 6796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:24.070223 6796 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:38415
I20260926 06:25:24.070254 6862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:38415 every 8 connection(s)
I20260926 06:25:24.070596 6796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:24.071221 6863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:24.073966 6863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347: Bootstrap starting.
I20260926 06:25:24.074479 6863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:24.074718 6863 log.cc:824] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:24.075347 6863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347: No bootstrap required, opened a new log
I20260926 06:25:24.076843 6863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e94bbad7d945a484217584d2d9b347" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38415 } }
I20260926 06:25:24.076977 6863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:24.076999 6863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86e94bbad7d945a484217584d2d9b347, State: Initialized, Role: FOLLOWER
I20260926 06:25:24.077138 6863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e94bbad7d945a484217584d2d9b347" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38415 } }
I20260926 06:25:24.077216 6863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:24.077239 6863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:24.077265 6863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:24.077740 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 6796
I20260926 06:25:24.077818 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:24.077850 6863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e94bbad7d945a484217584d2d9b347" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38415 } }
I20260926 06:25:24.077977 6863 leader_election.cc:302] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86e94bbad7d945a484217584d2d9b347; no voters:
I20260926 06:25:24.078047 2207 external_mini_cluster.cc:1428] Setting key 2eb49fdd27b90128f8565533740d73a5
I20260926 06:25:24.078153 6863 leader_election.cc:288] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:24.078202 6866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:24.078353 6866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [term 1 LEADER]: Becoming Leader. State: Replica: 86e94bbad7d945a484217584d2d9b347, State: Running, Role: LEADER
I20260926 06:25:24.078413 6863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:24.078464 6866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e94bbad7d945a484217584d2d9b347" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38415 } }
I20260926 06:25:24.078821 6866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86e94bbad7d945a484217584d2d9b347. Latest consensus state: current_term: 1 leader_uuid: "86e94bbad7d945a484217584d2d9b347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e94bbad7d945a484217584d2d9b347" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38415 } } }
I20260926 06:25:24.078902 6866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:24.079228 6867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86e94bbad7d945a484217584d2d9b347" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86e94bbad7d945a484217584d2d9b347" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38415 } } }
I20260926 06:25:24.079378 6867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:24.079994 6875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:24.080430 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.078675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.2.39.254:38415)
0926 06:25:24.078919 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.078928 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.078962 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:25:24.079082 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.079091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.079139 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.079298 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.079932 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.079936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.079936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.079941 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.079984 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.079999 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.080040 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.080177 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.080237 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":61,"threads_started":1}
I20260926 06:25:24.080698 6875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:24.082314 6875 catalog_manager.cc:1347] Generated new cluster ID: a66be57d722b4164a93d6df8eb8d8eeb
I20260926 06:25:24.082373 6875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:24.094324 6875 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:24.094522 6875 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:24.099536 6875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347: Generated new TSK 0
I20260926 06:25:24.099745 6875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:25:24.106383 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38415
--builtin_ntp_servers=127.2.39.212:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 06:25:24.192836 6887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:24.193068 6887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:24.193092 6887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:24.194339 6887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:24.194382 6887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:24.194409 6887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:24.195794 6887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:24.196067 6887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:24.196380 6887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:24.202266 6892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:24.202276 6895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:24.202276 6893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:24.202766 6887 server_base.cc:1034] running on GCE node
I20260926 06:25:24.202992 6887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:24.203243 6887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:24.204429 6887 hybrid_clock.cc:648] HybridClock initialized: now 1790403924204408 us; error 43 us; skew 500 ppm
I20260926 06:25:24.205825 6887 webserver.cc:460] Webserver started at http://127.2.39.193:36309/ using document root <none> and password file <none>
I20260926 06:25:24.206090 6887 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:24.206161 6887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:24.206315 6887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:24.207391 6887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "fe90d29da78a4a30b6d5a1c0017a61e8"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "c1fcac5b980bdbee4025b8f129d575c1"
server_key_iv: "94dd07f8b57c1263d2f9757daf51c050"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.207751 6887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "fe90d29da78a4a30b6d5a1c0017a61e8"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "c1fcac5b980bdbee4025b8f129d575c1"
server_key_iv: "94dd07f8b57c1263d2f9757daf51c050"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.209120 6887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:24.210065 6901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.210276 6887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:25:24.210387 6887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "fe90d29da78a4a30b6d5a1c0017a61e8"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "c1fcac5b980bdbee4025b8f129d575c1"
server_key_iv: "94dd07f8b57c1263d2f9757daf51c050"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.210492 6887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:24.213790 6887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:24.214107 6887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:24.214255 6887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:24.214465 6887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:24.214766 6887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:24.214820 6887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.214867 6887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:24.214895 6887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.220726 6887 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:36103
I20260926 06:25:24.220798 7014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:36103 every 8 connection(s)
I20260926 06:25:24.221128 6887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:24.221611 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 6887
I20260926 06:25:24.221720 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:24.221927 2207 external_mini_cluster.cc:1428] Setting key ebd68671b221f1c46a0f92db03ff5feb
I20260926 06:25:24.222889 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38415
--builtin_ntp_servers=127.2.39.212:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 06:25:24.224612 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.221925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:37969 (local address 127.2.39.254:38415)
0926 06:25:24.221975 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.221977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.222275 (+ 298us) server_negotiation.cc:407] Connection header received
0926 06:25:24.222881 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.222882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.222889 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.222935 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.224314 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.224315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.224315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.224319 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.224333 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.224338 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.224456 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.224548 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.224561 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:25:24.224860 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.221938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38415 (local address 127.2.39.193:37969)
0926 06:25:24.222155 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:24.222199 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:24.222782 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:24.223003 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:24.223006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:24.223016 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:24.224123 (+ 1107us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:24.224124 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:24.224163 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:24.224201 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:24.224469 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:24.224474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:24.224477 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:24.224557 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 06:25:24.224624 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260926 06:25:24.225432 7015 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38415
I20260926 06:25:24.225538 7015 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:24.225701 7015 heartbeater.cc:511] Master 127.2.39.254:38415 requested a full tablet report, sending...
I20260926 06:25:24.226214 6827 ts_manager.cc:194] Registered new tserver with Master: fe90d29da78a4a30b6d5a1c0017a61e8 (127.2.39.193:36103)
W20260926 06:25:24.303359 7018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:24.303570 7018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:24.303592 7018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:24.304755 7018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:24.304796 7018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:24.304821 7018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:24.306170 7018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:24.306412 7018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:24.306710 7018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:24.311625 7023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:24.311641 7026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:24.311683 7018 server_base.cc:1034] running on GCE node
W20260926 06:25:24.311641 7024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:24.312028 7018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:24.312284 7018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:24.313428 7018 hybrid_clock.cc:648] HybridClock initialized: now 1790403924313412 us; error 34 us; skew 500 ppm
I20260926 06:25:24.314733 7018 webserver.cc:460] Webserver started at http://127.2.39.194:32819/ using document root <none> and password file <none>
I20260926 06:25:24.314980 7018 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:24.315025 7018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:24.315181 7018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:24.316250 7018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "3179a615e81c4a87ad628e1b679a21b7"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "4f3be889bb65f7e6725a3e6173d7cc7f"
server_key_iv: "7af91efe1790c86587aa1ecb4faf6185"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.316623 7018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "3179a615e81c4a87ad628e1b679a21b7"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "4f3be889bb65f7e6725a3e6173d7cc7f"
server_key_iv: "7af91efe1790c86587aa1ecb4faf6185"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.317999 7018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:24.318699 7032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.318869 7018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.318969 7018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "3179a615e81c4a87ad628e1b679a21b7"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "4f3be889bb65f7e6725a3e6173d7cc7f"
server_key_iv: "7af91efe1790c86587aa1ecb4faf6185"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.319064 7018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:24.322479 7018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:24.322753 7018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:24.322894 7018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:24.323098 7018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:24.323397 7018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:24.323451 7018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.323493 7018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:24.323520 7018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.329128 7018 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:33519
I20260926 06:25:24.329174 7145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:33519 every 8 connection(s)
I20260926 06:25:24.329476 7018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:24.332170 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.330195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:44205 (local address 127.2.39.254:38415)
0926 06:25:24.330256 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.330258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.330481 (+ 223us) server_negotiation.cc:407] Connection header received
0926 06:25:24.330892 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.330893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.330897 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.330956 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.331999 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.332000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.332000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.332004 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.332018 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.332022 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.332068 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.332112 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.332122 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:25:24.332345 7148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.330180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38415 (local address 127.2.39.194:44205)
0926 06:25:24.330348 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:24.330392 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:24.330815 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:24.331004 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:24.331007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:24.331015 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:24.331869 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:24.331869 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:24.331895 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:24.331919 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:24.332065 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:24.332067 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:24.332069 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:24.332117 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:25:24.332160 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260926 06:25:24.332791 7146 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38415
I20260926 06:25:24.332890 7146 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:24.333051 7146 heartbeater.cc:511] Master 127.2.39.254:38415 requested a full tablet report, sending...
I20260926 06:25:24.333540 6827 ts_manager.cc:194] Registered new tserver with Master: 3179a615e81c4a87ad628e1b679a21b7 (127.2.39.194:33519)
I20260926 06:25:24.338158 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 7018
I20260926 06:25:24.338261 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:24.338486 2207 external_mini_cluster.cc:1428] Setting key 6511c2a3914fddcc5870144b59fde655
I20260926 06:25:24.339382 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38415
--builtin_ntp_servers=127.2.39.212:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 06:25:24.418329 7149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:24.418581 7149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:24.418619 7149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:24.419811 7149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:24.419876 7149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:24.419920 7149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:24.421303 7149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:24.421581 7149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:24.421923 7149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:24.426923 7157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:24.426973 7155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:24.426941 7154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:24.427254 7149 server_base.cc:1034] running on GCE node
I20260926 06:25:24.427497 7149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:24.427731 7149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:24.428889 7149 hybrid_clock.cc:648] HybridClock initialized: now 1790403924428878 us; error 45 us; skew 500 ppm
I20260926 06:25:24.430207 7149 webserver.cc:460] Webserver started at http://127.2.39.195:46413/ using document root <none> and password file <none>
I20260926 06:25:24.430452 7149 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:24.430523 7149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:24.430662 7149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:24.431700 7149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "cd307888f2824683810f5f5625e3233a"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "07e0b85634bcd1a896a9b0580751d17c"
server_key_iv: "eafad605c7397b5ece93c9446fa71b20"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.432068 7149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "cd307888f2824683810f5f5625e3233a"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "07e0b85634bcd1a896a9b0580751d17c"
server_key_iv: "eafad605c7397b5ece93c9446fa71b20"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.433321 7149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:25:24.434101 7163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.434304 7149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.434398 7149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "cd307888f2824683810f5f5625e3233a"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "07e0b85634bcd1a896a9b0580751d17c"
server_key_iv: "eafad605c7397b5ece93c9446fa71b20"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.434484 7149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:24.437711 7149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:24.437958 7149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:24.438107 7149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:24.438305 7149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:24.438591 7149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:24.438643 7149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.438683 7149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:24.438709 7149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.444319 7149 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:46463
I20260926 06:25:24.444370 7276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:46463 every 8 connection(s)
I20260926 06:25:24.444670 7149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:24.447748 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.445480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:53983 (local address 127.2.39.254:38415)
0926 06:25:24.445557 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.445559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.445833 (+ 274us) server_negotiation.cc:407] Connection header received
0926 06:25:24.446293 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.446294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.446298 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.446410 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.447528 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.447529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.447530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.447534 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.447549 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.447553 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.447614 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.447684 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.447696 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 06:25:24.447945 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.445486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38415 (local address 127.2.39.195:53983)
0926 06:25:24.445730 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:24.445761 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:24.446182 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:24.446472 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:24.446475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:24.446483 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:24.447383 (+ 900us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:24.447383 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:24.447409 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:24.447434 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:24.447620 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:24.447623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:24.447626 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:24.447682 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:25:24.447727 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":134,"threads_started":1}
I20260926 06:25:24.448433 7277 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38415
I20260926 06:25:24.448526 7277 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:24.448647 7277 heartbeater.cc:511] Master 127.2.39.254:38415 requested a full tablet report, sending...
I20260926 06:25:24.449000 6827 ts_manager.cc:194] Registered new tserver with Master: cd307888f2824683810f5f5625e3233a (127.2.39.195:46463)
I20260926 06:25:24.453845 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 7149
I20260926 06:25:24.453938 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:24.454088 2207 external_mini_cluster.cc:1428] Setting key 2dca927c1e96fb82bc839a722d7bfb56
I20260926 06:25:24.454998 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:25:24.457321 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.456387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.2.39.254:38415)
0926 06:25:24.456451 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.456452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.456544 (+ 92us) server_negotiation.cc:407] Connection header received
0926 06:25:24.456596 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.456596 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.456601 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.456660 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.457148 (+ 488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.457149 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.457150 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.457153 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.457169 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.457173 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.457226 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.457259 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.457272 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:25:24.458979 6827 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:24.460500 6827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:24.464461 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.462944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:33519 (local address 127.0.0.1:59768)
0926 06:25:24.463137 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:24.463147 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:24.463193 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:24.463440 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:24.463440 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:24.463442 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:24.463988 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:24.463989 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:24.464003 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:24.464037 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:24.464287 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:24.464292 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:24.464295 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:24.464357 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:25:24.464370 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":56,"threads_started":1}
I20260926 06:25:24.464576 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.462753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:36103 (local address 127.0.0.1:56730)
0926 06:25:24.463045 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:24.463055 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:24.463099 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:24.463577 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:24.463578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:24.463581 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:24.464174 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:24.464174 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:24.464180 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:24.464188 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:24.464388 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:24.464389 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:24.464390 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:25:24.464441 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:25:24.464456 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":84,"threads_started":1}
I20260926 06:25:24.464607 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.462779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33412 (local address 127.2.39.195:46463)
0926 06:25:24.463125 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.463129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.463140 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:25:24.463165 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.463169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.463178 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.463277 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.464189 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.464195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.464195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.464199 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.464250 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.464276 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.464341 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.464497 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.464514 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":90,"threads_started":1}
I20260926 06:25:24.464461 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.462720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:46463 (local address 127.0.0.1:33412)
0926 06:25:24.462908 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:24.462949 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:24.463003 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:24.463287 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:24.463293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:24.463300 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:24.464014 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:24.464014 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:24.464028 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:24.464040 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:24.464334 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:24.464335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:24.464335 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 06:25:24.464395 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 06:25:24.464406 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
I20260926 06:25:24.464906 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.463124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.2.39.194:33519)
0926 06:25:24.463294 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.463298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.463306 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:24.463331 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.463333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.463341 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.463639 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.464137 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.464145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.464145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.464150 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.464184 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.464210 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.464292 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.464781 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.464803 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260926 06:25:24.464908 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.462861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56730 (local address 127.2.39.193:36103)
0926 06:25:24.463033 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.463038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.463154 (+ 116us) server_negotiation.cc:407] Connection header received
0926 06:25:24.463180 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.463182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.463191 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.463257 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.464266 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.464273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.464273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.464277 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.464307 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.464334 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.464392 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.464446 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.464461 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260926 06:25:24.465416 7080 tablet_service.cc:1467] Processing CreateTablet for tablet 5679d3c99935473fab1bfe5c6676b21f (DEFAULT_TABLE table=test-table [id=7112f193de83446e9fde8d541463ff75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:24.465416 7211 tablet_service.cc:1467] Processing CreateTablet for tablet 5679d3c99935473fab1bfe5c6676b21f (DEFAULT_TABLE table=test-table [id=7112f193de83446e9fde8d541463ff75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:24.465531 6949 tablet_service.cc:1467] Processing CreateTablet for tablet 5679d3c99935473fab1bfe5c6676b21f (DEFAULT_TABLE table=test-table [id=7112f193de83446e9fde8d541463ff75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:24.465792 7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5679d3c99935473fab1bfe5c6676b21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:24.465792 7211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5679d3c99935473fab1bfe5c6676b21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:24.465901 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5679d3c99935473fab1bfe5c6676b21f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:24.468348 7291 tablet_bootstrap.cc:492] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8: Bootstrap starting.
I20260926 06:25:24.468976 7293 tablet_bootstrap.cc:492] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: Bootstrap starting.
I20260926 06:25:24.469159 7291 tablet_bootstrap.cc:654] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:24.469497 7291 log.cc:824] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:24.469657 7293 tablet_bootstrap.cc:654] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:24.469740 7292 tablet_bootstrap.cc:492] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: Bootstrap starting.
I20260926 06:25:24.469980 7293 log.cc:824] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:24.470312 7292 tablet_bootstrap.cc:654] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:24.470690 7292 log.cc:824] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:24.470700 7293 tablet_bootstrap.cc:492] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: No bootstrap required, opened a new log
I20260926 06:25:24.470795 7293 ts_tablet_manager.cc:1397] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 06:25:24.471436 7292 tablet_bootstrap.cc:492] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: No bootstrap required, opened a new log
I20260926 06:25:24.471518 7292 ts_tablet_manager.cc:1397] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:24.472908 7293 raft_consensus.cc:348] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 } } peers { permanent_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 36103 } } peers { permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 } }
I20260926 06:25:24.473048 7293 raft_consensus.cc:374] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:24.473070 7293 raft_consensus.cc:729] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd307888f2824683810f5f5625e3233a, State: Initialized, Role: FOLLOWER
I20260926 06:25:24.473191 7293 consensus_queue.cc:260] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 } } peers { permanent_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 36103 } } peers { permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 } }
I20260926 06:25:24.473460 7293 ts_tablet_manager.cc:1428] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:25:24.473477 7277 heartbeater.cc:503] Master 127.2.39.254:38415 was elected leader, sending a full tablet report...
I20260926 06:25:24.473444 7292 raft_consensus.cc:348] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 } } peers { permanent_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 36103 } } peers { permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 } }
I20260926 06:25:24.473623 7292 raft_consensus.cc:374] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:24.473663 7292 raft_consensus.cc:729] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3179a615e81c4a87ad628e1b679a21b7, State: Initialized, Role: FOLLOWER
I20260926 06:25:24.473771 7292 consensus_queue.cc:260] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 } } peers { permanent_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 36103 } } peers { permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 } }
I20260926 06:25:24.474025 7291 tablet_bootstrap.cc:492] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8: No bootstrap required, opened a new log
I20260926 06:25:24.474025 7292 ts_tablet_manager.cc:1428] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:25:24.474080 7291 ts_tablet_manager.cc:1397] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 06:25:24.474236 7146 heartbeater.cc:503] Master 127.2.39.254:38415 was elected leader, sending a full tablet report...
I20260926 06:25:24.476102 7291 raft_consensus.cc:348] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 } } peers { permanent_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 36103 } } peers { permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 } }
I20260926 06:25:24.476208 7291 raft_consensus.cc:374] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:24.476228 7291 raft_consensus.cc:729] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe90d29da78a4a30b6d5a1c0017a61e8, State: Initialized, Role: FOLLOWER
I20260926 06:25:24.476300 7291 consensus_queue.cc:260] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 } } peers { permanent_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 36103 } } peers { permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 } }
I20260926 06:25:24.476465 7291 ts_tablet_manager.cc:1428] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:25:24.476531 7015 heartbeater.cc:503] Master 127.2.39.254:38415 was elected leader, sending a full tablet report...
W20260926 06:25:24.579994 7147 tablet.cc:2369] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:25:24.599953 7299 raft_consensus.cc:482] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:24.600077 7299 raft_consensus.cc:504] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 } } peers { permanent_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 36103 } } peers { permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 } }
I20260926 06:25:24.600518 7299 leader_election.cc:288] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd307888f2824683810f5f5625e3233a (127.2.39.195:46463), 3179a615e81c4a87ad628e1b679a21b7 (127.2.39.194:33519)
I20260926 06:25:24.602097 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.600727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:46463 (local address 127.2.39.193:44263)
0926 06:25:24.600771 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:24.600782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:24.600822 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:24.601132 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:24.601132 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:24.601134 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:24.601726 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:24.601727 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:24.601733 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:24.601743 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:24.601945 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:24.601946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:24.601946 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 06:25:24.602002 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:25:24.602015 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 06:25:24.602113 7287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.600802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:44263 (local address 127.2.39.195:46463)
0926 06:25:24.600859 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.600861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.600880 (+ 19us) server_negotiation.cc:407] Connection header received
0926 06:25:24.600911 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.600912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.600919 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.601000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.601823 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.601825 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.601825 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.601830 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.601845 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.601849 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.601939 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.602025 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.602041 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:25:24.602164 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.600806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:45295 (local address 127.2.39.194:33519)
0926 06:25:24.600850 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.600852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.601001 (+ 149us) server_negotiation.cc:407] Connection header received
0926 06:25:24.601044 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.601044 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.601049 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.601086 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.601935 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.601936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.601937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.601941 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.601958 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.601963 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.602019 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.602073 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.602086 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:25:24.602207 7300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.600731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:33519 (local address 127.2.39.193:45295)
0926 06:25:24.600925 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:24.600935 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:24.600988 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:24.601107 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:24.601108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:24.601110 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:24.601818 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:24.601819 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:24.601825 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:24.601838 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:24.602024 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:24.602025 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:24.602025 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 06:25:24.602073 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:25:24.602085 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":48,"thread_start_us":83,"threads_started":1}
I20260926 06:25:24.602634 7231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5679d3c99935473fab1bfe5c6676b21f" candidate_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd307888f2824683810f5f5625e3233a" is_pre_election: true
I20260926 06:25:24.602681 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5679d3c99935473fab1bfe5c6676b21f" candidate_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3179a615e81c4a87ad628e1b679a21b7" is_pre_election: true
I20260926 06:25:24.602792 7231 raft_consensus.cc:2454] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe90d29da78a4a30b6d5a1c0017a61e8 in term 0.
I20260926 06:25:24.602810 7100 raft_consensus.cc:2454] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe90d29da78a4a30b6d5a1c0017a61e8 in term 0.
I20260926 06:25:24.603055 6905 leader_election.cc:302] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd307888f2824683810f5f5625e3233a, fe90d29da78a4a30b6d5a1c0017a61e8; no voters:
I20260926 06:25:24.603223 7299 raft_consensus.cc:2781] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:24.603260 7299 raft_consensus.cc:482] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:24.603282 7299 raft_consensus.cc:3037] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:24.603950 7299 raft_consensus.cc:504] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 } } peers { permanent_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 36103 } } peers { permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 } }
I20260926 06:25:24.604079 7299 leader_election.cc:288] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [CANDIDATE]: Term 1 election: Requested vote from peers cd307888f2824683810f5f5625e3233a (127.2.39.195:46463), 3179a615e81c4a87ad628e1b679a21b7 (127.2.39.194:33519)
I20260926 06:25:24.604242 7100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5679d3c99935473fab1bfe5c6676b21f" candidate_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3179a615e81c4a87ad628e1b679a21b7"
I20260926 06:25:24.604235 7231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5679d3c99935473fab1bfe5c6676b21f" candidate_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd307888f2824683810f5f5625e3233a"
I20260926 06:25:24.604305 7100 raft_consensus.cc:3037] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:24.604327 7231 raft_consensus.cc:3037] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:24.604997 7100 raft_consensus.cc:2454] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe90d29da78a4a30b6d5a1c0017a61e8 in term 1.
I20260926 06:25:24.605016 7231 raft_consensus.cc:2454] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe90d29da78a4a30b6d5a1c0017a61e8 in term 1.
I20260926 06:25:24.605191 6904 leader_election.cc:302] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3179a615e81c4a87ad628e1b679a21b7, fe90d29da78a4a30b6d5a1c0017a61e8; no voters:
I20260926 06:25:24.605299 7299 raft_consensus.cc:2781] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:24.605439 7299 raft_consensus.cc:686] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 1 LEADER]: Becoming Leader. State: Replica: fe90d29da78a4a30b6d5a1c0017a61e8, State: Running, Role: LEADER
I20260926 06:25:24.605538 7299 consensus_queue.cc:237] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 } } peers { permanent_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 36103 } } peers { permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 } }
I20260926 06:25:24.606400 6827 catalog_manager.cc:5495] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe90d29da78a4a30b6d5a1c0017a61e8 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe90d29da78a4a30b6d5a1c0017a61e8" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 36103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:24.613087 7288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.611984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.2.39.193:36103)
0926 06:25:24.612048 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.612050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.612057 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:24.612086 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.612087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.612099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.612172 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.612824 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.612826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.612826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.612865 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.612879 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.612885 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.612956 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.613005 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.613020 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:25:24.614909 7100 raft_consensus.cc:1261] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7 [term 1 FOLLOWER]: Refusing update from remote peer fe90d29da78a4a30b6d5a1c0017a61e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:24.614921 7231 raft_consensus.cc:1261] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a [term 1 FOLLOWER]: Refusing update from remote peer fe90d29da78a4a30b6d5a1c0017a61e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:24.615275 7301 consensus_queue.cc:1035] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd307888f2824683810f5f5625e3233a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:24.615406 7299 consensus_queue.cc:1035] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3179a615e81c4a87ad628e1b679a21b7" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:24.617789 7304 mvcc.cc:205] Tried to move back new op lower bound from 7333494475219935232 to 7333494475185008640. Current Snapshot: MvccSnapshot[applied={T|T < 7333494475219935232}]
I20260926 06:25:24.617893 7305 mvcc.cc:205] Tried to move back new op lower bound from 7333494475219935232 to 7333494475185008640. Current Snapshot: MvccSnapshot[applied={T|T < 7333494475219935232}]
I20260926 06:25:24.619856 6826 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:24.619951 6826 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:24.620743 6826 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86e94bbad7d945a484217584d2d9b347: Sending DeleteTablet for 3 replicas of tablet 5679d3c99935473fab1bfe5c6676b21f
I20260926 06:25:24.621129 7080 tablet_service.cc:1514] Processing DeleteTablet for tablet 5679d3c99935473fab1bfe5c6676b21f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:24 UTC) from {username='slave'} at 127.0.0.1:59768
I20260926 06:25:24.621119 6949 tablet_service.cc:1514] Processing DeleteTablet for tablet 5679d3c99935473fab1bfe5c6676b21f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:24 UTC) from {username='slave'} at 127.0.0.1:56730
I20260926 06:25:24.621361 7315 tablet_replica.cc:330] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8: stopping tablet replica
I20260926 06:25:24.621356 7211 tablet_service.cc:1514] Processing DeleteTablet for tablet 5679d3c99935473fab1bfe5c6676b21f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:24 UTC) from {username='slave'} at 127.0.0.1:33412
I20260926 06:25:24.621457 7315 raft_consensus.cc:2229] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:24.621599 7315 raft_consensus.cc:2258] T 5679d3c99935473fab1bfe5c6676b21f P fe90d29da78a4a30b6d5a1c0017a61e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:24.621626 7316 tablet_replica.cc:330] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: stopping tablet replica
I20260926 06:25:24.621752 7314 tablet_replica.cc:330] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: stopping tablet replica
I20260926 06:25:24.621811 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 6887
I20260926 06:25:24.621829 7314 raft_consensus.cc:2229] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:24.622785 7314 raft_consensus.cc:2258] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:24.622782 7316 raft_consensus.cc:2229] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:24.623059 7316 raft_consensus.cc:2258] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:24.623464 7316 ts_tablet_manager.cc:1905] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:24.623631 7314 ts_tablet_manager.cc:1905] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:24.625209 7316 ts_tablet_manager.cc:1918] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:24.625286 7316 log.cc:1196] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/5679d3c99935473fab1bfe5c6676b21f
I20260926 06:25:24.625504 7314 ts_tablet_manager.cc:1918] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:24.625615 7314 log.cc:1196] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/5679d3c99935473fab1bfe5c6676b21f
I20260926 06:25:24.625626 7316 ts_tablet_manager.cc:1939] T 5679d3c99935473fab1bfe5c6676b21f P cd307888f2824683810f5f5625e3233a: Deleting consensus metadata
I20260926 06:25:24.625867 7314 ts_tablet_manager.cc:1939] T 5679d3c99935473fab1bfe5c6676b21f P 3179a615e81c4a87ad628e1b679a21b7: Deleting consensus metadata
I20260926 06:25:24.626034 6814 catalog_manager.cc:4841] TS cd307888f2824683810f5f5625e3233a (127.2.39.195:46463): tablet 5679d3c99935473fab1bfe5c6676b21f (table test-table [id=7112f193de83446e9fde8d541463ff75]) successfully deleted
I20260926 06:25:24.626287 6813 catalog_manager.cc:4841] TS 3179a615e81c4a87ad628e1b679a21b7 (127.2.39.194:33519): tablet 5679d3c99935473fab1bfe5c6676b21f (table test-table [id=7112f193de83446e9fde8d541463ff75]) successfully deleted
W20260926 06:25:24.627954 6813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:36103 (error 108)
I20260926 06:25:24.628177 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 7018
I20260926 06:25:24.628829 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.628282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:36103 (local address 127.0.0.1:56734)
0926 06:25:24.628759 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:24.628801 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:36103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459}
W20260926 06:25:24.628939 6813 catalog_manager.cc:4584] TS fe90d29da78a4a30b6d5a1c0017a61e8 (127.2.39.193:36103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5679d3c99935473fab1bfe5c6676b21f: Network error: Client connection negotiation failed: client connection to 127.2.39.193:36103: connect: Connection refused (error 111)
I20260926 06:25:24.632920 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 7149
I20260926 06:25:24.637777 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 6796
2026-09-26T06:25:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (700 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 06:25:24.656988 2207 mini_oidc.cc:152] Starting JWKS server
I20260926 06:25:24.658087 2207 webserver.cc:460] Webserver started at https://127.0.0.1:41905/ using document root <none> and password file <none>
I20260926 06:25:24.658166 2207 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 06:25:24.658291 2207 webserver.cc:460] Webserver started at http://127.0.0.1:34399/ using document root <none> and password file <none>
2026-09-26T06:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:24Z Disabled control of system clock
I20260926 06:25:24.665904 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:35701
--webserver_interface=127.2.39.254
--webserver_port=0
--builtin_ntp_servers=127.2.39.212:39019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:35701
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41905/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/testchainca.cert with env {}
W20260926 06:25:24.745348 7323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:24.745563 7323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:24.745603 7323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:24.746727 7323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:24.746762 7323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:24.746775 7323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:24.746788 7323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:24.746799 7323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:25:24.746811 7323 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:25:24.746822 7323 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41905/jwks.json
I20260926 06:25:24.748284 7323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:39019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:35701
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:35701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41905/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:24.748481 7323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:24.748636 7323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:24.753482 7331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:24.753515 7329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:24.753484 7328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:24.753703 7323 server_base.cc:1034] running on GCE node
I20260926 06:25:24.753942 7323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:24.754186 7323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:24.755353 7323 hybrid_clock.cc:648] HybridClock initialized: now 1790403924755336 us; error 35 us; skew 500 ppm
I20260926 06:25:24.756588 7323 webserver.cc:460] Webserver started at http://127.2.39.254:38197/ using document root <none> and password file <none>
I20260926 06:25:24.756834 7323 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:24.756902 7323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:24.757050 7323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:24.758162 7323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "bac20a4ff141453c9727e11a94800260"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "bf6bc95cda4055db4c810c98aaf667e2"
server_key_iv: "aeb09b8bfad1d1fcd26a3ab7ab0d5b7b"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.758525 7323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "bac20a4ff141453c9727e11a94800260"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "bf6bc95cda4055db4c810c98aaf667e2"
server_key_iv: "aeb09b8bfad1d1fcd26a3ab7ab0d5b7b"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.759780 7323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:24.760512 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.760700 7323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:24.760787 7323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "bac20a4ff141453c9727e11a94800260"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "bf6bc95cda4055db4c810c98aaf667e2"
server_key_iv: "aeb09b8bfad1d1fcd26a3ab7ab0d5b7b"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.760862 7323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:24.785036 7323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:24.785353 7323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:24.785506 7323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:24.789394 7323 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:35701
I20260926 06:25:24.789472 7389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:35701 every 8 connection(s)
I20260926 06:25:24.789817 7323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:24.790737 7390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:24.792930 7390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260: Bootstrap starting.
I20260926 06:25:24.793452 7390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:24.793771 7390 log.cc:824] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:24.794384 7390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260: No bootstrap required, opened a new log
I20260926 06:25:24.795876 7390 raft_consensus.cc:348] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac20a4ff141453c9727e11a94800260" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35701 } }
I20260926 06:25:24.796005 7390 raft_consensus.cc:374] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:24.796044 7390 raft_consensus.cc:729] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac20a4ff141453c9727e11a94800260, State: Initialized, Role: FOLLOWER
I20260926 06:25:24.796145 7390 consensus_queue.cc:260] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac20a4ff141453c9727e11a94800260" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35701 } }
I20260926 06:25:24.796231 7390 raft_consensus.cc:388] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:24.796267 7390 raft_consensus.cc:482] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:24.796314 7390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:24.796806 7390 raft_consensus.cc:504] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac20a4ff141453c9727e11a94800260" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35701 } }
I20260926 06:25:24.796923 7390 leader_election.cc:302] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bac20a4ff141453c9727e11a94800260; no voters:
I20260926 06:25:24.797123 7390 leader_election.cc:288] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:24.797181 7393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:24.797370 7390 sys_catalog.cc:564] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:24.797432 7393 raft_consensus.cc:686] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [term 1 LEADER]: Becoming Leader. State: Replica: bac20a4ff141453c9727e11a94800260, State: Running, Role: LEADER
I20260926 06:25:24.797545 7393 consensus_queue.cc:237] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac20a4ff141453c9727e11a94800260" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35701 } }
I20260926 06:25:24.797969 7395 sys_catalog.cc:455] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bac20a4ff141453c9727e11a94800260. Latest consensus state: current_term: 1 leader_uuid: "bac20a4ff141453c9727e11a94800260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac20a4ff141453c9727e11a94800260" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35701 } } }
I20260926 06:25:24.798053 7395 sys_catalog.cc:458] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:24.798040 7394 sys_catalog.cc:455] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bac20a4ff141453c9727e11a94800260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac20a4ff141453c9727e11a94800260" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 35701 } } }
I20260926 06:25:24.798130 7394 sys_catalog.cc:458] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:24.798287 7402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:24.798851 7402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:24.800048 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 7323
I20260926 06:25:24.800168 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:24.800336 2207 external_mini_cluster.cc:1428] Setting key 9541e376f06a7ff166ab26b280dc4dc8
I20260926 06:25:24.800420 7402 catalog_manager.cc:1347] Generated new cluster ID: edefaa8ba1ba404fab9ceadddf2ccb75
I20260926 06:25:24.800460 7402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:24.806380 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.800925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.2.39.254:35701)
0926 06:25:24.801103 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.801112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.801143 (+ 31us) server_negotiation.cc:407] Connection header received
0926 06:25:24.801252 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.801259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.801323 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.801477 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:24.801697 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:24.803041 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:24.804191 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:24.804817 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:24.804952 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:24.805687 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:24.805692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:24.805692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:24.805892 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:24.805929 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:24.805942 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:24.806002 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:24.806118 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 06:25:24.806192 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260926 06:25:24.807340 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:25:24.813299 7402 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:24.813956 7402 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:24.819437 7402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bac20a4ff141453c9727e11a94800260: Generated new TSK 0
I20260926 06:25:24.819621 7402 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 06:25:24.844897 7420 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 06:25:24.845003 7420 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 06:25:24.845266 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:24.830247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.2.39.254:35701)
0926 06:25:24.830311 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:24.830314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:24.830456 (+ 142us) server_negotiation.cc:407] Connection header received
0926 06:25:24.830556 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:24.830557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:24.830567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:24.830624 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:25:24.830772 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:24.831271 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:24.832432 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:24.832605 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:24.832653 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:24.832865 (+ 212us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:25:24.845030 (+ 12165us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 06:25:24.845192 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 06:25:24.845325 7413 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 06:25:24.845654 7419 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:24.830183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:35701 (local address 127.0.0.1:45506)
0926 06:25:24.830381 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:24.830394 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:24.830467 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:24.830649 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:24.830650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:24.830652 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:25:24.830705 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:24.830707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:24.831351 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:24.831353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:24.832339 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:24.832342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:24.832676 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:24.832677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:24.832737 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:24.832759 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:25:24.845191 (+ 12432us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 06:25:24.845293 (+ 102us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.39.254:35701: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.39.254:35701: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 06:25:24.846107 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 7323
2026-09-26T06:25:24Z chronyd exiting
I20260926 06:25:24.881558 2207 test_util.cc:175] -----------------------------------------------
I20260926 06:25:24.881716 2207 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790403899360596-2207-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (225 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 06:25:24.882576 2207 mini_oidc.cc:152] Starting JWKS server
I20260926 06:25:24.883303 2207 webserver.cc:460] Webserver started at https://127.0.0.1:38955/ using document root <none> and password file <none>
I20260926 06:25:24.883378 2207 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 06:25:24.883581 2207 webserver.cc:460] Webserver started at http://127.0.0.1:41467/ using document root <none> and password file <none>
2026-09-26T06:25:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:24Z Disabled control of system clock
I20260926 06:25:24.891129 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:33067
--webserver_interface=127.2.39.254
--webserver_port=0
--builtin_ntp_servers=127.2.39.212:44389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:33067
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38955/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/testchainca.cert with env {}
W20260926 06:25:24.969311 7426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:24.969569 7426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:24.969602 7426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:24.970767 7426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:24.970837 7426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:24.970858 7426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:24.970881 7426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:24.970909 7426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:25:24.970932 7426 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:25:24.970959 7426 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38955/jwks.json
I20260926 06:25:24.972437 7426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:33067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:33067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38955/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:24.972712 7426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:24.972930 7426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:24.977604 7431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:24.977651 7434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:24.977705 7426 server_base.cc:1034] running on GCE node
W20260926 06:25:24.977650 7432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:24.978140 7426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:24.978454 7426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:24.979631 7426 hybrid_clock.cc:648] HybridClock initialized: now 1790403924979624 us; error 51 us; skew 500 ppm
I20260926 06:25:24.980801 7426 webserver.cc:460] Webserver started at http://127.2.39.254:45353/ using document root <none> and password file <none>
I20260926 06:25:24.981041 7426 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:24.981113 7426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:24.981243 7426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:24.982380 7426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "389b592c8fbf4f0cb81bdd66a1cda848"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "e794787956058f9298962c08250ccf1a"
server_key_iv: "f1a674cf8b529ea6c01ffdd06d2d73ad"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.982755 7426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "389b592c8fbf4f0cb81bdd66a1cda848"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "e794787956058f9298962c08250ccf1a"
server_key_iv: "f1a674cf8b529ea6c01ffdd06d2d73ad"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.984040 7426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:25:24.984822 7440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:24.984995 7426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:24.985095 7426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "389b592c8fbf4f0cb81bdd66a1cda848"
format_stamp: "Formatted at 2026-09-26 06:25:24 on dist-test-slave-fkbn"
server_key: "e794787956058f9298962c08250ccf1a"
server_key_iv: "f1a674cf8b529ea6c01ffdd06d2d73ad"
server_key_version: "encryptionkey@0"
I20260926 06:25:24.985180 7426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:24.995626 7426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:24.995908 7426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:24.996052 7426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:24.999943 7426 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:33067
I20260926 06:25:24.999971 7492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:33067 every 8 connection(s)
I20260926 06:25:25.000319 7426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:25.001281 7493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:25.003536 7493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848: Bootstrap starting.
I20260926 06:25:25.004067 7493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:25.004293 7493 log.cc:824] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:25.004964 7493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848: No bootstrap required, opened a new log
I20260926 06:25:25.005144 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 7426
I20260926 06:25:25.005257 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:25.005475 2207 external_mini_cluster.cc:1428] Setting key cdbe52537c2fa5b8b2bc06220f26e530
I20260926 06:25:25.007086 7493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389b592c8fbf4f0cb81bdd66a1cda848" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33067 } }
I20260926 06:25:25.007258 7493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:25.007293 7493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389b592c8fbf4f0cb81bdd66a1cda848, State: Initialized, Role: FOLLOWER
I20260926 06:25:25.007437 7493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389b592c8fbf4f0cb81bdd66a1cda848" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33067 } }
I20260926 06:25:25.007501 7493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:25.007524 7493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:25.007576 7493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:25.008309 7493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389b592c8fbf4f0cb81bdd66a1cda848" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33067 } }
I20260926 06:25:25.008419 7493 leader_election.cc:302] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 389b592c8fbf4f0cb81bdd66a1cda848; no voters:
I20260926 06:25:25.008558 7493 leader_election.cc:288] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:25.008644 7498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:25.008782 7498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [term 1 LEADER]: Becoming Leader. State: Replica: 389b592c8fbf4f0cb81bdd66a1cda848, State: Running, Role: LEADER
I20260926 06:25:25.008792 7493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:25.008903 7498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389b592c8fbf4f0cb81bdd66a1cda848" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33067 } }
I20260926 06:25:25.009671 7500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389b592c8fbf4f0cb81bdd66a1cda848. Latest consensus state: current_term: 1 leader_uuid: "389b592c8fbf4f0cb81bdd66a1cda848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389b592c8fbf4f0cb81bdd66a1cda848" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33067 } } }
I20260926 06:25:25.009755 7500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:25.009655 7499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "389b592c8fbf4f0cb81bdd66a1cda848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389b592c8fbf4f0cb81bdd66a1cda848" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33067 } } }
I20260926 06:25:25.009933 7499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:25.009985 7507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:25.010473 7507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:25.012144 7507 catalog_manager.cc:1347] Generated new cluster ID: 75edec9f3b264d1db199a548177828f1
I20260926 06:25:25.012249 7507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:25.012916 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:25.006032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.2.39.254:33067)
0926 06:25:25.006261 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:25.006270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:25.006303 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:25:25.007033 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:25.007043 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:25.007126 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:25.007279 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:25.007621 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.009284 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.010507 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.011112 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.011229 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:25.012220 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:25.012224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:25.012224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:25.012434 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:25.012470 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:25.012481 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:25.012558 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:25.012635 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 06:25:25.012714 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260926 06:25:25.013895 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:25:25.027633 7507 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:25.028193 7507 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:25.035848 7507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 389b592c8fbf4f0cb81bdd66a1cda848: Generated new TSK 0
I20260926 06:25:25.036068 7507 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 06:25:25.067776 7523 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 06:25:25.067864 7523 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 06:25:25.068197 7497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:25.045617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52320 (local address 127.2.39.254:33067)
0926 06:25:25.045678 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:25.045680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:25.045829 (+ 149us) server_negotiation.cc:407] Connection header received
0926 06:25:25.045902 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:25.045903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:25.045912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:25.045971 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:25:25.046098 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.046753 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.047973 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.048149 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.048208 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:25.048388 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:25:25.067948 (+ 19560us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 06:25:25.068139 (+ 191us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 06:25:25.068276 7497 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 06:25:25.068315 7522 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:25.045567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33067 (local address 127.0.0.1:52320)
0926 06:25:25.045758 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:25.045769 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:25.045835 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:25.045982 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:25.045983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:25.045985 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:25:25.046041 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:25.046043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.046869 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.046870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:25.047866 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:25.047869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.048229 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.048230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:25.048292 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:25.048297 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:25:25.068096 (+ 19799us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 06:25:25.068215 (+ 119us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.39.254:33067: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":128,"threads_started":1}
I20260926 06:25:25.068660 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 7426
2026-09-26T06:25:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (223 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 06:25:25.106119 2207 mini_oidc.cc:152] Starting JWKS server
I20260926 06:25:25.106823 2207 webserver.cc:460] Webserver started at https://127.0.0.1:43597/ using document root <none> and password file <none>
I20260926 06:25:25.106894 2207 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 06:25:25.107048 2207 webserver.cc:460] Webserver started at http://127.0.0.1:41085/ using document root <none> and password file <none>
2026-09-26T06:25:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:25Z Disabled control of system clock
I20260926 06:25:25.115352 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:39555
--webserver_interface=127.2.39.254
--webserver_port=0
--builtin_ntp_servers=127.2.39.212:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:39555
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43597/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 06:25:25.199429 7529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:25.199628 7529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:25.199651 7529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:25.200775 7529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:25.200812 7529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:25.200825 7529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:25.200838 7529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:25.200850 7529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:25:25.200862 7529 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:25:25.200873 7529 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43597/jwks.json
I20260926 06:25:25.202247 7529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:39555
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:39555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43597/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:25.202451 7529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:25.202680 7529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:25.207612 7537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:25.207631 7534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:25.207633 7535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:25.207937 7529 server_base.cc:1034] running on GCE node
I20260926 06:25:25.208258 7529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:25.208556 7529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:25.209743 7529 hybrid_clock.cc:648] HybridClock initialized: now 1790403925209736 us; error 36 us; skew 500 ppm
I20260926 06:25:25.211179 7529 webserver.cc:460] Webserver started at http://127.2.39.254:35585/ using document root <none> and password file <none>
I20260926 06:25:25.211442 7529 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:25.211510 7529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:25.211661 7529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:25.212757 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "12c4b2f41a464667b510de5719a6edd2"
format_stamp: "Formatted at 2026-09-26 06:25:25 on dist-test-slave-fkbn"
server_key: "777aef454a1821a588a1e1bd62711f5a"
server_key_iv: "31241358fe73218899705b9e9b5b43ef"
server_key_version: "encryptionkey@0"
I20260926 06:25:25.213140 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "12c4b2f41a464667b510de5719a6edd2"
format_stamp: "Formatted at 2026-09-26 06:25:25 on dist-test-slave-fkbn"
server_key: "777aef454a1821a588a1e1bd62711f5a"
server_key_iv: "31241358fe73218899705b9e9b5b43ef"
server_key_version: "encryptionkey@0"
I20260926 06:25:25.214593 7529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:25.215371 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:25.215570 7529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:25.215688 7529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "12c4b2f41a464667b510de5719a6edd2"
format_stamp: "Formatted at 2026-09-26 06:25:25 on dist-test-slave-fkbn"
server_key: "777aef454a1821a588a1e1bd62711f5a"
server_key_iv: "31241358fe73218899705b9e9b5b43ef"
server_key_version: "encryptionkey@0"
I20260926 06:25:25.215764 7529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:25.231141 7529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:25.231436 7529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:25.231586 7529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:25.235206 7529 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:39555
I20260926 06:25:25.235247 7595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:39555 every 8 connection(s)
I20260926 06:25:25.235606 7529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:25.236281 7596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:25.238860 7596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2: Bootstrap starting.
I20260926 06:25:25.239427 7596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:25.239719 7596 log.cc:824] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:25.239966 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 7529
I20260926 06:25:25.240072 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:25.240280 2207 external_mini_cluster.cc:1428] Setting key 5d50c56f60320b8fa28bcb97485b3570
I20260926 06:25:25.240576 7596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2: No bootstrap required, opened a new log
I20260926 06:25:25.242456 7596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4b2f41a464667b510de5719a6edd2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39555 } }
I20260926 06:25:25.242607 7596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:25.242650 7596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c4b2f41a464667b510de5719a6edd2, State: Initialized, Role: FOLLOWER
I20260926 06:25:25.242781 7596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4b2f41a464667b510de5719a6edd2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39555 } }
I20260926 06:25:25.242885 7596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:25.242955 7596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:25.243009 7596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:25.243542 7596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4b2f41a464667b510de5719a6edd2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39555 } }
I20260926 06:25:25.243626 7596 leader_election.cc:302] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12c4b2f41a464667b510de5719a6edd2; no voters:
I20260926 06:25:25.243820 7596 leader_election.cc:288] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:25.243866 7601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:25.243996 7601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [term 1 LEADER]: Becoming Leader. State: Replica: 12c4b2f41a464667b510de5719a6edd2, State: Running, Role: LEADER
I20260926 06:25:25.244093 7596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:25.244081 7601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4b2f41a464667b510de5719a6edd2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39555 } }
I20260926 06:25:25.244369 7601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c4b2f41a464667b510de5719a6edd2. Latest consensus state: current_term: 1 leader_uuid: "12c4b2f41a464667b510de5719a6edd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4b2f41a464667b510de5719a6edd2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39555 } } }
I20260926 06:25:25.244428 7601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:25.244704 7607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:25.245198 7607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:25.244401 7602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12c4b2f41a464667b510de5719a6edd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12c4b2f41a464667b510de5719a6edd2" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39555 } } }
I20260926 06:25:25.245885 7602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:25.250589 7607 catalog_manager.cc:1347] Generated new cluster ID: 73d41e547a6140d8a87c2cf3710e72d4
I20260926 06:25:25.250646 7607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:25.253626 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:25.240992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.2.39.254:39555)
0926 06:25:25.241207 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:25.241214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:25.241238 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:25:25.242439 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:25.242446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:25.242504 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:25.242652 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:25.242855 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.244500 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.245692 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.246325 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.246451 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:25.252890 (+ 6439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:25.252895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:25.252895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:25.253114 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:25.253159 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:25.253173 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:25.253248 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:25.253317 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 06:25:25.253396 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
I20260926 06:25:25.254539 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:25:25.257490 7607 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:25.258111 7607 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:25.264760 7607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12c4b2f41a464667b510de5719a6edd2: Generated new TSK 0
I20260926 06:25:25.264930 7607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:25:25.281755 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:25.277810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.2.39.254:39555)
0926 06:25:25.277875 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:25.277878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:25.278037 (+ 159us) server_negotiation.cc:407] Connection header received
0926 06:25:25.278108 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:25.278108 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:25.278120 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:25.278187 (+ 67us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:25:25.278316 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.278873 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.280013 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.280387 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.280462 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:25.280656 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:25:25.281495 (+ 839us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43597/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 06:25:25.281681 (+ 186us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43597/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 06:25:25.281828 7625 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:25.277751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:39555 (local address 127.0.0.1:39346)
0926 06:25:25.277954 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:25.277965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:25.278038 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:25.278196 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:25.278197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:25.278199 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:25:25.278257 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:25.278259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.278963 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.278964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:25.279910 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:25.279913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.280484 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.280485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:25.280550 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:25.280555 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:25:25.281624 (+ 1069us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43597/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 06:25:25.281722 (+ 98us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.39.254:39555: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43597/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":136,"threads_started":1}
W20260926 06:25:25.281837 7600 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43597/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260926 06:25:25.282045 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 7529
E20260926 06:25:25.296094 7626 webserver.cc:533] Webserver: sslize failed:
E20260926 06:25:25.296180 7626 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T06:25:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T06:25:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:25Z Disabled control of system clock
I20260926 06:25:25.326908 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:46813
--webserver_interface=127.2.39.254
--webserver_port=0
--builtin_ntp_servers=127.2.39.212:41191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:46813
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/testchainca.cert with env {}
W20260926 06:25:25.406248 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:25.406450 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:25.406473 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:25.407668 7630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:25.407713 7630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:25.407725 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:25.407738 7630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:25.407790 7630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 06:25:25.407809 7630 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 06:25:25.407837 7630 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 06:25:25.409411 7630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:41191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:46813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:46813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:25.409685 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:25.409922 7630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:25.414851 7635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:25.414851 7636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:25.414906 7638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:25.415225 7630 server_base.cc:1034] running on GCE node
I20260926 06:25:25.415469 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:25.415711 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:25.416890 7630 hybrid_clock.cc:648] HybridClock initialized: now 1790403925416872 us; error 29 us; skew 500 ppm
I20260926 06:25:25.418288 7630 webserver.cc:460] Webserver started at http://127.2.39.254:43307/ using document root <none> and password file <none>
I20260926 06:25:25.418556 7630 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:25.418607 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:25.418733 7630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:25.419854 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "55bcbb40fa0f45cf901e52e94d6fbd85"
format_stamp: "Formatted at 2026-09-26 06:25:25 on dist-test-slave-fkbn"
server_key: "fdb5fd17ddd8eb704f45ce7958a28a0a"
server_key_iv: "4d8fb69d2600f3c898bdd8b65bb13eb4"
server_key_version: "encryptionkey@0"
I20260926 06:25:25.420238 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "55bcbb40fa0f45cf901e52e94d6fbd85"
format_stamp: "Formatted at 2026-09-26 06:25:25 on dist-test-slave-fkbn"
server_key: "fdb5fd17ddd8eb704f45ce7958a28a0a"
server_key_iv: "4d8fb69d2600f3c898bdd8b65bb13eb4"
server_key_version: "encryptionkey@0"
I20260926 06:25:25.421670 7630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 06:25:25.422552 7644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:25.422763 7630 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:25.422873 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "55bcbb40fa0f45cf901e52e94d6fbd85"
format_stamp: "Formatted at 2026-09-26 06:25:25 on dist-test-slave-fkbn"
server_key: "fdb5fd17ddd8eb704f45ce7958a28a0a"
server_key_iv: "4d8fb69d2600f3c898bdd8b65bb13eb4"
server_key_version: "encryptionkey@0"
I20260926 06:25:25.422943 7630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:25.439711 7630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:25.440004 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:25.440155 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:25.443897 7630 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:46813
I20260926 06:25:25.443933 7696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:46813 every 8 connection(s)
I20260926 06:25:25.444275 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:25.445151 7697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:25.447362 7697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85: Bootstrap starting.
I20260926 06:25:25.447868 7697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:25.448103 7697 log.cc:824] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:25.448711 7697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85: No bootstrap required, opened a new log
I20260926 06:25:25.450344 7697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bcbb40fa0f45cf901e52e94d6fbd85" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46813 } }
I20260926 06:25:25.450479 7697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:25.450501 7697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55bcbb40fa0f45cf901e52e94d6fbd85, State: Initialized, Role: FOLLOWER
I20260926 06:25:25.450601 7697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bcbb40fa0f45cf901e52e94d6fbd85" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46813 } }
I20260926 06:25:25.450650 7697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:25.450670 7697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:25.450696 7697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:25.451144 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 7630
I20260926 06:25:25.451258 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:25.451320 7697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bcbb40fa0f45cf901e52e94d6fbd85" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46813 } }
I20260926 06:25:25.451428 7697 leader_election.cc:302] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55bcbb40fa0f45cf901e52e94d6fbd85; no voters:
I20260926 06:25:25.451457 2207 external_mini_cluster.cc:1428] Setting key d79fd73df7f2c15a656fe4537288a020
I20260926 06:25:25.451587 7697 leader_election.cc:288] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:25.451622 7700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:25.451757 7700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [term 1 LEADER]: Becoming Leader. State: Replica: 55bcbb40fa0f45cf901e52e94d6fbd85, State: Running, Role: LEADER
I20260926 06:25:25.451824 7697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:25.451867 7700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bcbb40fa0f45cf901e52e94d6fbd85" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46813 } }
I20260926 06:25:25.452404 7701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55bcbb40fa0f45cf901e52e94d6fbd85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bcbb40fa0f45cf901e52e94d6fbd85" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46813 } } }
I20260926 06:25:25.452490 7701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:25.452652 7702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55bcbb40fa0f45cf901e52e94d6fbd85. Latest consensus state: current_term: 1 leader_uuid: "55bcbb40fa0f45cf901e52e94d6fbd85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55bcbb40fa0f45cf901e52e94d6fbd85" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46813 } } }
I20260926 06:25:25.452772 7702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:25.452925 7711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:25.453406 7711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:25.455592 7711 catalog_manager.cc:1347] Generated new cluster ID: fd40427d85ba46ee9b3dfb5a93811c6a
I20260926 06:25:25.455646 7711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:25.458129 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:25.452060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.2.39.254:46813)
0926 06:25:25.452968 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:25.452977 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:25.453017 (+ 40us) server_negotiation.cc:407] Connection header received
0926 06:25:25.453142 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:25.453149 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:25.453229 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:25.453375 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:25.453714 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.455040 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.456003 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.456625 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.456753 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:25.457409 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:25.457413 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:25.457413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:25.457648 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:25.457692 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:25.457709 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:25.457780 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:25.457846 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 06:25:25.457924 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":812,"thread_start_us":64,"threads_started":1}
I20260926 06:25:25.459049 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:25:25.462719 7711 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:25.463196 7711 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:25.475039 7711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55bcbb40fa0f45cf901e52e94d6fbd85: Generated new TSK 0
I20260926 06:25:25.475227 7711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:25:25.505195 7704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:25.479854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.2.39.254:46813)
0926 06:25:25.479892 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:25.479898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:25.479995 (+ 97us) server_negotiation.cc:407] Connection header received
0926 06:25:25.480074 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:25.480075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:25.480084 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:25.480157 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0926 06:25:25.480288 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.480774 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.481909 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.482113 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.482166 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:25.482338 (+ 172us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 06:25:25.504910 (+ 22572us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 06:25:25.505115 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53136: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 06:25:25.505276 7704 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53136: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 06:25:25.505254 7726 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:25.479803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46813 (local address 127.0.0.1:53136)
0926 06:25:25.479945 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:25.479955 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:25.480011 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:25.480166 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:25.480167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:25.480169 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 06:25:25.480227 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:25.480229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.480844 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.480845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:25.481831 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:25.481834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:25.482188 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:25.482189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:25.482249 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:25.482254 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 06:25:25.505062 (+ 22808us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 06:25:25.505178 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.39.254:46813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260926 06:25:25.505674 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 7630
2026-09-26T06:25:25Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (202 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (89 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:25 dist-test-slave-fkbn krb5kdc[7738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:25 dist-test-slave-fkbn krb5kdc[7738](info): set up 2 sockets
Sep 26 06:25:25 dist-test-slave-fkbn krb5kdc[7738](info): commencing operation
krb5kdc: starting...
W20260926 06:25:26.714193 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:26 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:26.832055 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:34971
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:34971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:26.909847 7751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:26.910075 7751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:26.910112 7751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:26.911301 7751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:26.911369 7751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:26.911393 7751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:26.911414 7751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:26.911443 7751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:26.912845 7751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:34971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:34971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:26.913113 7751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:26.913326 7751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:26.917997 7757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:26.918037 7756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:26.918148 7751 server_base.cc:1034] running on GCE node
W20260926 06:25:26.918099 7759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:26.918479 7751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:26.918778 7751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:26.919942 7751 hybrid_clock.cc:648] HybridClock initialized: now 1790403926919927 us; error 27 us; skew 500 ppm
Sep 26 06:25:26 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403926, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:26.922381 7751 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:26.922845 7751 webserver.cc:460] Webserver started at http://127.2.39.254:40061/ using document root <none> and password file <none>
I20260926 06:25:26.923079 7751 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:26.923144 7751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:26.923295 7751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:26.924279 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff"
format_stamp: "Formatted at 2026-09-26 06:25:26 on dist-test-slave-fkbn"
server_key: "95e9e27c8de3abb5c9d30d7a1aeb3e60"
server_key_iv: "ff82a49ae7a7e5ecbc6fad5ce26c424b"
server_key_version: "encryptionkey@0"
I20260926 06:25:26.924635 7751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff"
format_stamp: "Formatted at 2026-09-26 06:25:26 on dist-test-slave-fkbn"
server_key: "95e9e27c8de3abb5c9d30d7a1aeb3e60"
server_key_iv: "ff82a49ae7a7e5ecbc6fad5ce26c424b"
server_key_version: "encryptionkey@0"
I20260926 06:25:26.926028 7751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 06:25:26.926718 7766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:26.926877 7751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:26.926973 7751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff"
format_stamp: "Formatted at 2026-09-26 06:25:26 on dist-test-slave-fkbn"
server_key: "95e9e27c8de3abb5c9d30d7a1aeb3e60"
server_key_iv: "ff82a49ae7a7e5ecbc6fad5ce26c424b"
server_key_version: "encryptionkey@0"
I20260926 06:25:26.927049 7751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:26.936120 7751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:26.938017 7751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:26.938167 7751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:26.942083 7751 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:34971
I20260926 06:25:26.942128 7818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:34971 every 8 connection(s)
I20260926 06:25:26.942461 7751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:26.943289 7819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:26.945477 7819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff: Bootstrap starting.
I20260926 06:25:26.946072 7819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:26.946235 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 7751
I20260926 06:25:26.946352 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:26.946439 7819 log.cc:824] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:26.946553 2207 external_mini_cluster.cc:1428] Setting key bfc3c856a7c9819fe3f9275030c1144a
I20260926 06:25:26.947198 7819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff: No bootstrap required, opened a new log
I20260926 06:25:26.949543 7819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34971 } }
I20260926 06:25:26.949792 7819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:26.949846 7819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93cc6d83bca846fab68fe8bfb2b7f7ff, State: Initialized, Role: FOLLOWER
I20260926 06:25:26.949991 7819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34971 } }
I20260926 06:25:26.950098 7819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:26.950145 7819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:26.950197 7819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:26.950769 7819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34971 } }
I20260926 06:25:26.950896 7819 leader_election.cc:302] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93cc6d83bca846fab68fe8bfb2b7f7ff; no voters:
I20260926 06:25:26.951098 7819 leader_election.cc:288] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:26.951166 7824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:26.951303 7824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [term 1 LEADER]: Becoming Leader. State: Replica: 93cc6d83bca846fab68fe8bfb2b7f7ff, State: Running, Role: LEADER
I20260926 06:25:26.951365 7819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:26.951388 7824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34971 } }
I20260926 06:25:26.951756 7824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93cc6d83bca846fab68fe8bfb2b7f7ff. Latest consensus state: current_term: 1 leader_uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34971 } } }
I20260926 06:25:26.951822 7824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:26.951776 7825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93cc6d83bca846fab68fe8bfb2b7f7ff" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34971 } } }
I20260926 06:25:26.951898 7825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:26.952239 7830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:26.952765 7830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 06:25:26 dist-test-slave-fkbn krb5kdc[7738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:26.954445 7830 catalog_manager.cc:1347] Generated new cluster ID: 698be7aec65d4219979e9f26b8619901
I20260926 06:25:26.954499 7830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:26.957237 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:26.947052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.2.39.254:34971)
0926 06:25:26.947222 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:26.947232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:26.947270 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:25:26.947705 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:26.947711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:26.947756 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:26.947901 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:26.948517 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:26.949410 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:26.950330 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:26.950732 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:26.950867 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:26.953716 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:26.953719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:26.953720 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:26.953732 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:26.955798 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:26.956164 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:26.956167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:26.956167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:26.956250 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:26.956437 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:26.956438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:26.956438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:26.956664 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:26.956771 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:26.956872 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 06:25:26.956976 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260926 06:25:26.970716 7830 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:26.971397 7830 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:26.976941 7830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93cc6d83bca846fab68fe8bfb2b7f7ff: Generated new TSK 0
I20260926 06:25:26.977101 7830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:27.006631 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34971
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:27.086966 7845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:27.087181 7845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:27.087218 7845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:27.088327 7845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:27.088395 7845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:27.088438 7845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:27.089946 7845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:27.090197 7845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:27.090463 7845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:27.095294 7851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:27.095289 7853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:27.095289 7850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:27.095602 7845 server_base.cc:1034] running on GCE node
I20260926 06:25:27.095783 7845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:27.096040 7845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:27.097180 7845 hybrid_clock.cc:648] HybridClock initialized: now 1790403927097167 us; error 27 us; skew 500 ppm
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:27.099614 7845 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:27.100075 7845 webserver.cc:460] Webserver started at http://127.2.39.193:40767/ using document root <none> and password file <none>
I20260926 06:25:27.100306 7845 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:27.100349 7845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:27.100517 7845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:27.101536 7845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "8001012237be423c9d3aa0ec0b53f842"
format_stamp: "Formatted at 2026-09-26 06:25:27 on dist-test-slave-fkbn"
server_key: "55c3095e09f196c904fa8dcefe00b348"
server_key_iv: "04b0289fec3c2fe6ff4244407fdf5ec4"
server_key_version: "encryptionkey@0"
I20260926 06:25:27.101944 7845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "8001012237be423c9d3aa0ec0b53f842"
format_stamp: "Formatted at 2026-09-26 06:25:27 on dist-test-slave-fkbn"
server_key: "55c3095e09f196c904fa8dcefe00b348"
server_key_iv: "04b0289fec3c2fe6ff4244407fdf5ec4"
server_key_version: "encryptionkey@0"
I20260926 06:25:27.103186 7845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:27.103891 7860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:27.104074 7845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:27.104171 7845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "8001012237be423c9d3aa0ec0b53f842"
format_stamp: "Formatted at 2026-09-26 06:25:27 on dist-test-slave-fkbn"
server_key: "55c3095e09f196c904fa8dcefe00b348"
server_key_iv: "04b0289fec3c2fe6ff4244407fdf5ec4"
server_key_version: "encryptionkey@0"
I20260926 06:25:27.104250 7845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:27.120837 7845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:27.122723 7845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:27.122885 7845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:27.123135 7845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:27.123435 7845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:27.123490 7845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:27.123538 7845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:27.123565 7845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:27.129217 7845 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:40391
I20260926 06:25:27.129254 7973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:40391 every 8 connection(s)
I20260926 06:25:27.129638 7845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:27.131219 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 7845
I20260926 06:25:27.131341 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:27.131516 2207 external_mini_cluster.cc:1428] Setting key 7fe9237423dbbce32ed0a7e4d42a9962
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
I20260926 06:25:27.140211 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:27.130429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:35781 (local address 127.2.39.254:34971)
0926 06:25:27.130471 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:27.130473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:27.130710 (+ 237us) server_negotiation.cc:407] Connection header received
0926 06:25:27.131764 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:27.131765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:27.131777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:27.131834 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:27.132459 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.132985 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.134590 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.134794 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.134884 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:27.137279 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:27.137280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:27.137281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:27.137288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:27.138762 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.139248 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.139249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.139249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.139307 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.139559 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.139560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.139561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.139677 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:27.139753 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:27.140024 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 06:25:27.140116 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:25:27.140409 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:27.130375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:34971 (local address 127.2.39.193:35781)
0926 06:25:27.130572 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:27.130617 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:27.131653 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:27.131905 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:27.131910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:27.132161 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:27.132348 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:27.132351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.133080 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.133081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:27.134489 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:27.134492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.134906 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.134907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:27.135047 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:27.135565 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:27.135580 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:27.137183 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:27.138864 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:27.138870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:27.138876 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:27.139164 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:27.139373 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:27.139374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:27.139375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:27.139487 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:27.139744 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:27.139750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:27.139889 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 06:25:27.140005 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 06:25:27.140168 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260926 06:25:27.140933 7974 heartbeater.cc:347] Connected to a master server at 127.2.39.254:34971
I20260926 06:25:27.141047 7974 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:27.141230 7974 heartbeater.cc:511] Master 127.2.39.254:34971 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
I20260926 06:25:27.141817 7783 ts_manager.cc:194] Registered new tserver with Master: 8001012237be423c9d3aa0ec0b53f842 (127.2.39.193:40391)
I20260926 06:25:27.142444 7783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:35781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:27.156590 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34971
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:27.236289 7979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:27.236474 7979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:27.236495 7979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:27.237711 7979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:27.237752 7979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:27.237777 7979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:27.239233 7979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:27.239454 7979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:27.239686 7979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:27.244433 7985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:27.244491 7987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:27.244454 7984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:27.244676 7979 server_base.cc:1034] running on GCE node
I20260926 06:25:27.244882 7979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:27.245118 7979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:27.246274 7979 hybrid_clock.cc:648] HybridClock initialized: now 1790403927246259 us; error 26 us; skew 500 ppm
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:27.248623 7979 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:27.249053 7979 webserver.cc:460] Webserver started at http://127.2.39.194:41089/ using document root <none> and password file <none>
I20260926 06:25:27.249243 7979 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:27.249282 7979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:27.249367 7979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:27.250367 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "39430fd3cbf542c3bc1f7428e17d1008"
format_stamp: "Formatted at 2026-09-26 06:25:27 on dist-test-slave-fkbn"
server_key: "252481417a8e448aaa403019fe8a8c15"
server_key_iv: "6741a2e206bdd40d07c1cff308f70447"
server_key_version: "encryptionkey@0"
I20260926 06:25:27.250651 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "39430fd3cbf542c3bc1f7428e17d1008"
format_stamp: "Formatted at 2026-09-26 06:25:27 on dist-test-slave-fkbn"
server_key: "252481417a8e448aaa403019fe8a8c15"
server_key_iv: "6741a2e206bdd40d07c1cff308f70447"
server_key_version: "encryptionkey@0"
I20260926 06:25:27.251916 7979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:25:27.252689 7994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:27.252846 7979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:27.252910 7979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "39430fd3cbf542c3bc1f7428e17d1008"
format_stamp: "Formatted at 2026-09-26 06:25:27 on dist-test-slave-fkbn"
server_key: "252481417a8e448aaa403019fe8a8c15"
server_key_iv: "6741a2e206bdd40d07c1cff308f70447"
server_key_version: "encryptionkey@0"
I20260926 06:25:27.252986 7979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:27.265329 7979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:27.267122 7979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:27.267270 7979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:27.267510 7979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:27.267794 7979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:27.267827 7979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:27.267848 7979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:27.267894 7979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:27.273700 7979 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:43331
I20260926 06:25:27.273746 8107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:43331 every 8 connection(s)
I20260926 06:25:27.274042 7979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:27.281288 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 7979
I20260926 06:25:27.281371 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:27.281558 2207 external_mini_cluster.cc:1428] Setting key 0f0eab6b50a46ea0806a1a33d4a0a63f
I20260926 06:25:27.283509 8110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:27.274744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:34971 (local address 127.2.39.194:53853)
0926 06:25:27.274889 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:27.274920 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:27.275802 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:27.276046 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:27.276050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:27.276285 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:27.276468 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:27.276471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.277245 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.277246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:27.278069 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:27.278071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.278454 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.278455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:27.278554 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:27.278972 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:27.278980 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:27.280258 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:27.282068 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:27.282073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:27.282075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:27.282386 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:27.282665 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:27.282665 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:27.282665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:27.282762 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:27.283058 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:27.283061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:27.283166 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 06:25:27.283245 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:25:27.283355 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
I20260926 06:25:27.283514 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:27.274753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:53853 (local address 127.2.39.254:34971)
0926 06:25:27.274803 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:27.274805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:27.275015 (+ 210us) server_negotiation.cc:407] Connection header received
0926 06:25:27.275921 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:27.275923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:27.275933 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:27.275987 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:27.276580 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.277153 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.278165 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.278375 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.278442 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:27.280338 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:27.280339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:27.280339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:27.280346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:27.281998 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.282507 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.282508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.282509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.282582 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.282831 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.282839 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.282839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.282989 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:27.283063 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:27.283289 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 06:25:27.283376 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:25:27.284057 8108 heartbeater.cc:347] Connected to a master server at 127.2.39.254:34971
I20260926 06:25:27.284173 8108 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:27.284345 8108 heartbeater.cc:511] Master 127.2.39.254:34971 requested a full tablet report, sending...
I20260926 06:25:27.284745 7783 ts_manager.cc:194] Registered new tserver with Master: 39430fd3cbf542c3bc1f7428e17d1008 (127.2.39.194:43331)
I20260926 06:25:27.285174 7783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:53853
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:27.307832 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34971
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:27.386865 8113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:27.387079 8113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:27.387113 8113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:27.388190 8113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:27.388252 8113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:27.388293 8113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:27.389694 8113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34971
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:27.389933 8113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:27.390137 8113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:27.395133 8119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:27.395146 8118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:27.395190 8121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:27.395473 8113 server_base.cc:1034] running on GCE node
I20260926 06:25:27.395658 8113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:27.395903 8113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:27.397056 8113 hybrid_clock.cc:648] HybridClock initialized: now 1790403927397045 us; error 22 us; skew 500 ppm
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:27.399395 8113 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:27.399871 8113 webserver.cc:460] Webserver started at http://127.2.39.195:39873/ using document root <none> and password file <none>
I20260926 06:25:27.400086 8113 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:27.400156 8113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:27.400301 8113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:27.401291 8113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "bd8b296cea33436491f4405692d66988"
format_stamp: "Formatted at 2026-09-26 06:25:27 on dist-test-slave-fkbn"
server_key: "a3010a3b07a79dc11f862e12c2739701"
server_key_iv: "1fb1838ab67b568365ece802a22020c1"
server_key_version: "encryptionkey@0"
I20260926 06:25:27.401711 8113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "bd8b296cea33436491f4405692d66988"
format_stamp: "Formatted at 2026-09-26 06:25:27 on dist-test-slave-fkbn"
server_key: "a3010a3b07a79dc11f862e12c2739701"
server_key_iv: "1fb1838ab67b568365ece802a22020c1"
server_key_version: "encryptionkey@0"
I20260926 06:25:27.403049 8113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:27.403766 8128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:27.403972 8113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:27.404066 8113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "bd8b296cea33436491f4405692d66988"
format_stamp: "Formatted at 2026-09-26 06:25:27 on dist-test-slave-fkbn"
server_key: "a3010a3b07a79dc11f862e12c2739701"
server_key_iv: "1fb1838ab67b568365ece802a22020c1"
server_key_version: "encryptionkey@0"
I20260926 06:25:27.404151 8113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:27.429277 8113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:27.431317 8113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:27.431473 8113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:27.431710 8113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:27.432013 8113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:27.432070 8113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:27.432112 8113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:27.432140 8113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:27.437860 8113 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:36753
I20260926 06:25:27.437902 8241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:36753 every 8 connection(s)
I20260926 06:25:27.438239 8113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:27.442550 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 8113
I20260926 06:25:27.442631 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:27.442837 2207 external_mini_cluster.cc:1428] Setting key 892b20112d8db7eb35ac0438e859bd2b
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:27.447386 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:27.439004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:37809 (local address 127.2.39.254:34971)
0926 06:25:27.439048 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:27.439050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:27.439249 (+ 199us) server_negotiation.cc:407] Connection header received
0926 06:25:27.440169 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:27.440170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:27.440179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:27.440242 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:27.440791 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.441410 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.442455 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.442702 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.442764 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:27.444937 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:27.444938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:27.444938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:27.444945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:27.446173 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.446571 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.446572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.446572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.446620 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.446828 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.446828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.446828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.446953 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:27.447001 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:27.447258 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 06:25:27.447313 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:25:27.447645 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:27.438894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:34971 (local address 127.2.39.195:37809)
0926 06:25:27.439145 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:27.439175 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:27.440074 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:27.440296 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:27.440299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:27.440538 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:27.440719 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:27.440723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.441491 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.441491 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:27.442321 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:27.442323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.442775 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.442776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:27.442918 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:27.443485 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:27.443494 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:27.444852 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:27.446242 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:27.446247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:27.446249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:27.446506 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:27.446677 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:27.446678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:27.446678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:27.446778 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:27.447017 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:27.447022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:27.447149 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 06:25:27.447233 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 06:25:27.447354 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":105,"threads_started":1}
I20260926 06:25:27.448073 8242 heartbeater.cc:347] Connected to a master server at 127.2.39.254:34971
I20260926 06:25:27.448171 8242 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:27.448345 8242 heartbeater.cc:511] Master 127.2.39.254:34971 requested a full tablet report, sending...
I20260926 06:25:27.448735 7783 ts_manager.cc:194] Registered new tserver with Master: bd8b296cea33436491f4405692d66988 (127.2.39.195:36753)
I20260926 06:25:27.449113 7783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:37809
I20260926 06:25:27.454596 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/test-admin.keytab.
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:27.469954 2207 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 06:25:27.471556 8249 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:27.470757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.2.39.193:40391)
0926 06:25:27.470954 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:27.470958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:27.470966 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:27.470989 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:27.470990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:27.471003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:27.471088 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:27.471468 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55210: BlockingRecv error: recv got EOF from 127.0.0.1:55210 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":95,"threads_started":1}
W20260926 06:25:27.471658 2207 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5cc' of type FILE
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:27.472632 2207 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:27.473551 2207 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:25:27.481068 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:27.474068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.2.39.193:40391)
0926 06:25:27.474126 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:27.474128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:27.474171 (+ 43us) server_negotiation.cc:407] Connection header received
0926 06:25:27.474235 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:27.474236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:27.474247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:27.474295 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:27.474671 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.475375 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.476090 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.476406 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.476497 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:27.478629 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:27.478636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:27.478637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:27.478644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:27.479938 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.480290 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.480293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.480293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.480354 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.480549 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.480549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.480550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.480676 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:27.480817 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:27.480886 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 06:25:27.480958 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 06:25:27.482672 8249 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:27.482151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.2.39.193:40391)
0926 06:25:27.482186 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:27.482187 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:27.482213 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:25:27.482280 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:27.482281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:27.482291 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:27.482343 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:27.482622 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55230: BlockingRecv error: recv got EOF from 127.0.0.1:55230 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 06:25:27.482733 2207 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5cc' of type FILE
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:27.483592 2207 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:27.484447 2207 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:25:27.490729 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:27.484857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.2.39.193:40391)
0926 06:25:27.484912 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:27.484914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:27.484967 (+ 53us) server_negotiation.cc:407] Connection header received
0926 06:25:27.485001 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:27.485001 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:27.485009 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:27.485057 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:27.485374 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.485856 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.486512 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.486691 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.486738 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:27.488511 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:27.488513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:27.488513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:27.488519 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:27.489809 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.490132 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.490133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.490134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.490186 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.490382 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.490383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.490383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.490470 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:27.490539 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:27.490621 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 06:25:27.490678 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 06:25:27.492372 8249 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:27.491818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55244 (local address 127.2.39.193:40391)
0926 06:25:27.491874 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:27.491876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:27.491925 (+ 49us) server_negotiation.cc:407] Connection header received
0926 06:25:27.491995 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:27.491996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:27.492004 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:27.492075 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:27.492332 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55244: BlockingRecv error: recv got EOF from 127.0.0.1:55244 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 06:25:27.492614 2207 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790403899360596-2207-0/krb5kdc/krb5cc' of type FILE
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:27.493605 2207 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:27.494472 2207 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[7738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:25:27.500730 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:27.494953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55256 (local address 127.2.39.193:40391)
0926 06:25:27.494985 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:27.494991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:27.494998 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:27.495048 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:27.495048 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:27.495055 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:27.495112 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:27.495424 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.495897 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.496503 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:27.496673 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:27.496726 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:27.498633 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:27.498634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:27.498634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:27.498640 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:27.499801 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.500109 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.500109 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.500110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.500152 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:27.500314 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:27.500314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:27.500314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:27.500392 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:27.500455 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:27.500572 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 06:25:27.500635 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 06:25:27.501425 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 7845
I20260926 06:25:27.506371 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 7979
I20260926 06:25:27.511065 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 8113
I20260926 06:25:27.516322 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 7751
2026-09-26T06:25:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1837 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[8262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[8262](info): set up 2 sockets
Sep 26 06:25:27 dist-test-slave-fkbn krb5kdc[8262](info): commencing operation
krb5kdc: starting...
W20260926 06:25:28.653808 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:28 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.254.keytab.
Entry for principal oryx/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.254.keytab.
I20260926 06:25:28.772853 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:33879
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.254.keytab
--principal=oryx/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:33879
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:28.855814 8275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:28.856009 8275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:28.856030 8275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:28.857162 8275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:28.857203 8275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:28.857216 8275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:28.857229 8275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:28.857241 8275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:28.858669 8275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:33879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.254.keytab
--principal=oryx/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:33879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:28.858846 8275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:28.859002 8275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:28.863775 8283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:28.863809 8275 server_base.cc:1034] running on GCE node
W20260926 06:25:28.863780 8281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:28.863785 8280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:28.864239 8275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:28.864519 8275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:28.865644 8275 hybrid_clock.cc:648] HybridClock initialized: now 1790403928865635 us; error 32 us; skew 500 ppm
Sep 26 06:25:28 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403928, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:28.867991 8275 init.cc:377] Logged in from keytab as oryx/127.2.39.254@KRBTEST.COM (short username oryx)
I20260926 06:25:28.868450 8275 webserver.cc:460] Webserver started at http://127.2.39.254:42141/ using document root <none> and password file <none>
I20260926 06:25:28.868671 8275 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:28.868748 8275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:28.868891 8275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:28.869977 8275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "3e4964c5316c40d9b58b83994c8db608"
format_stamp: "Formatted at 2026-09-26 06:25:28 on dist-test-slave-fkbn"
server_key: "42fe678ca3f95b540e7c522be1723646"
server_key_iv: "d88ba498b3c0db10838e2b6cfa798de5"
server_key_version: "encryptionkey@0"
I20260926 06:25:28.870343 8275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "3e4964c5316c40d9b58b83994c8db608"
format_stamp: "Formatted at 2026-09-26 06:25:28 on dist-test-slave-fkbn"
server_key: "42fe678ca3f95b540e7c522be1723646"
server_key_iv: "d88ba498b3c0db10838e2b6cfa798de5"
server_key_version: "encryptionkey@0"
I20260926 06:25:28.871642 8275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:28.872349 8290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:28.872525 8275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:28.872624 8275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "3e4964c5316c40d9b58b83994c8db608"
format_stamp: "Formatted at 2026-09-26 06:25:28 on dist-test-slave-fkbn"
server_key: "42fe678ca3f95b540e7c522be1723646"
server_key_iv: "d88ba498b3c0db10838e2b6cfa798de5"
server_key_version: "encryptionkey@0"
I20260926 06:25:28.872699 8275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:28.890630 8275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:28.892544 8275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:28.892705 8275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:28.896924 8275 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:33879
I20260926 06:25:28.896991 8342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:33879 every 8 connection(s)
I20260926 06:25:28.897339 8275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:28.897472 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 8275
I20260926 06:25:28.897627 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:28.897850 2207 external_mini_cluster.cc:1428] Setting key 68d44da689d3717e24567801cb581c6c
I20260926 06:25:28.898591 8343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:28.901381 8343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608: Bootstrap starting.
I20260926 06:25:28.902047 8343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:28.902350 8343 log.cc:824] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:28.903040 8343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608: No bootstrap required, opened a new log
Sep 26 06:25:28 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:28.905175 8343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } }
I20260926 06:25:28.905298 8343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:28.905344 8343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e4964c5316c40d9b58b83994c8db608, State: Initialized, Role: FOLLOWER
I20260926 06:25:28.905486 8343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } }
I20260926 06:25:28.905608 8343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:28.905650 8343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:28.905700 8343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:28.906203 8343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } }
I20260926 06:25:28.906281 8343 leader_election.cc:302] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e4964c5316c40d9b58b83994c8db608; no voters:
I20260926 06:25:28.906456 8343 leader_election.cc:288] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:28.906510 8348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:28.906699 8343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:28.906725 8348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 1 LEADER]: Becoming Leader. State: Replica: 3e4964c5316c40d9b58b83994c8db608, State: Running, Role: LEADER
I20260926 06:25:28.906967 8348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } }
I20260926 06:25:28.907330 8349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } }
I20260926 06:25:28.907413 8349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:28.907618 8350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e4964c5316c40d9b58b83994c8db608. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } }
I20260926 06:25:28.907752 8350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:28.907913 8357 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:28.908442 8357 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:28.908883 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:28.898383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47870 (local address 127.2.39.254:33879)
0926 06:25:28.898553 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:28.898562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:28.898592 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:25:28.899065 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:28.899072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:28.899117 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:28.899226 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:28.899969 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:28.900820 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:28.901878 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:28.902328 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:28.902405 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:28.905051 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:28.905054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:28.905054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:28.905063 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:28.907152 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:28.907610 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:28.907613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:28.907614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:28.907750 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:28.907997 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:28.907998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:28.907998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:28.908297 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:28.908445 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:28.908566 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 06:25:28.908696 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260926 06:25:28.909881 8357 catalog_manager.cc:1347] Generated new cluster ID: 9b4400c3851c4f079ff38b0eaf864b50
I20260926 06:25:28.909933 8357 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:28.928141 8357 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:28.928684 8357 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:28.945459 8357 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608: Generated new TSK 0
I20260926 06:25:28.945720 8357 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab.
Entry for principal oryx/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab.
I20260926 06:25:28.972669 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33879
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab
--principal=oryx/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:29.052403 8369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:29.052611 8369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:29.052634 8369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:29.053726 8369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:29.053761 8369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:29.053786 8369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:29.055143 8369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab
--principal=oryx/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:29.055316 8369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:29.055485 8369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:29.060326 8375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.060333 8377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.060367 8374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:29.060495 8369 server_base.cc:1034] running on GCE node
I20260926 06:25:29.060702 8369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:29.060938 8369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:29.062095 8369 hybrid_clock.cc:648] HybridClock initialized: now 1790403929062073 us; error 35 us; skew 500 ppm
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:29.064445 8369 init.cc:377] Logged in from keytab as oryx/127.2.39.193@KRBTEST.COM (short username oryx)
I20260926 06:25:29.064890 8369 webserver.cc:460] Webserver started at http://127.2.39.193:40411/ using document root <none> and password file <none>
I20260926 06:25:29.065111 8369 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:29.065178 8369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:29.065325 8369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:29.066421 8369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "57c3d1ab035f49e488da5c43d18427cc"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "71ee776d440bc71766bdcd3c88c89e70"
server_key_iv: "542b6e60c7a048cb22a544930c658d1b"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.066784 8369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "57c3d1ab035f49e488da5c43d18427cc"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "71ee776d440bc71766bdcd3c88c89e70"
server_key_iv: "542b6e60c7a048cb22a544930c658d1b"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.068017 8369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:25:29.068678 8384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.068869 8369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:29.068962 8369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "57c3d1ab035f49e488da5c43d18427cc"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "71ee776d440bc71766bdcd3c88c89e70"
server_key_iv: "542b6e60c7a048cb22a544930c658d1b"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.069047 8369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:29.078625 8369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:29.080286 8369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:29.080443 8369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:29.080658 8369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:29.080961 8369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:29.081018 8369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.081053 8369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:29.081084 8369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.086813 8369 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:46281
I20260926 06:25:29.086870 8497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:46281 every 8 connection(s)
I20260926 06:25:29.087148 8369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:29.096971 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.087935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:45485 (local address 127.2.39.254:33879)
0926 06:25:29.088043 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.088046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.088219 (+ 173us) server_negotiation.cc:407] Connection header received
0926 06:25:29.088774 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.088775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.088788 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.088849 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.089901 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.090416 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.091473 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.091676 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.091767 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.093994 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.093996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.093996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.094005 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.095521 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.095985 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.095987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.095987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.096051 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.096304 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.096305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.096305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.096433 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.096516 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.096777 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.096872 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 06:25:29.097148 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.087860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.193:45485)
0926 06:25:29.088074 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.088118 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.088655 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.088920 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.088925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.089501 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.089764 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.089769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.090542 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.090543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.091362 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.091365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.091775 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.091777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.091920 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.092470 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.092477 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.093893 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.095649 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.095656 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.095658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.095891 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.096122 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.096124 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.096124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.096232 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.096507 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.096513 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.096645 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.096757 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.096918 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260926 06:25:29.097271 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 8369
I20260926 06:25:29.097409 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:29.097682 2207 external_mini_cluster.cc:1428] Setting key 5bc45d476e21ed3d4c97e716a2e2b45a
I20260926 06:25:29.097795 8498 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33879
I20260926 06:25:29.097895 8498 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:29.098079 8498 heartbeater.cc:511] Master 127.2.39.254:33879 requested a full tablet report, sending...
I20260926 06:25:29.098606 8307 ts_manager.cc:194] Registered new tserver with Master: 57c3d1ab035f49e488da5c43d18427cc (127.2.39.193:46281)
I20260926 06:25:29.099258 8307 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:45485
WARNING: no policy specified for oryx/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab.
Entry for principal oryx/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab.
I20260926 06:25:29.120594 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33879
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab
--principal=oryx/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:29.200809 8503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:29.201004 8503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:29.201027 8503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:29.202148 8503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:29.202188 8503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:29.202214 8503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:29.203533 8503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab
--principal=oryx/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:29.203702 8503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:29.203857 8503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:29.208792 8509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.208798 8511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:29.208814 8503 server_base.cc:1034] running on GCE node
W20260926 06:25:29.208801 8508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:29.209270 8503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:29.209478 8503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:29.210678 8503 hybrid_clock.cc:648] HybridClock initialized: now 1790403929210666 us; error 28 us; skew 500 ppm
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:29.213135 8503 init.cc:377] Logged in from keytab as oryx/127.2.39.194@KRBTEST.COM (short username oryx)
I20260926 06:25:29.213658 8503 webserver.cc:460] Webserver started at http://127.2.39.194:45131/ using document root <none> and password file <none>
I20260926 06:25:29.213886 8503 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:29.213950 8503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:29.214088 8503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:29.215070 8503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "1f2184cfa81daad61100068d12d92da3"
server_key_iv: "ae352c28eaed95e359f6642cedd3a69c"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.215436 8503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "1f2184cfa81daad61100068d12d92da3"
server_key_iv: "ae352c28eaed95e359f6642cedd3a69c"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.216693 8503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:29.217423 8518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.217618 8503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:29.217718 8503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "1f2184cfa81daad61100068d12d92da3"
server_key_iv: "ae352c28eaed95e359f6642cedd3a69c"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.217804 8503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:29.232932 8503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:29.234925 8503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:29.235102 8503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:29.235316 8503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:29.235610 8503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:29.235664 8503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.235702 8503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:29.235730 8503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.241575 8503 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:38443
I20260926 06:25:29.241636 8631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:38443 every 8 connection(s)
I20260926 06:25:29.241930 8503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:29.245049 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 8503
I20260926 06:25:29.245174 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:29.245363 2207 external_mini_cluster.cc:1428] Setting key 350baee5823780fc3b2a2ca738f30789
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:29.251974 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.242604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:39695 (local address 127.2.39.254:33879)
0926 06:25:29.242669 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.242686 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.242892 (+ 206us) server_negotiation.cc:407] Connection header received
0926 06:25:29.243498 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.243500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.243512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.243564 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.244491 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.245034 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.246208 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.246433 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.246521 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.248950 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.248951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.248952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.248958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.250368 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.250963 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.250964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.250965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.251027 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.251354 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.251355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.251355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.251463 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.251534 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.251843 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.251909 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:25:29.252238 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.242574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.194:39695)
0926 06:25:29.242749 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.242795 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.243386 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.243641 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.243645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.244200 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.244391 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.244395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.245129 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.245130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.246116 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.246119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.246528 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.246529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.246661 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.247183 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.247193 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.248860 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.250451 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.250458 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.250461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.250807 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.251120 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.251121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.251121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.251267 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.251536 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.251542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.251703 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.251824 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.252010 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260926 06:25:29.252835 8632 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33879
I20260926 06:25:29.252954 8632 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:29.253154 8632 heartbeater.cc:511] Master 127.2.39.254:33879 requested a full tablet report, sending...
I20260926 06:25:29.253592 8307 ts_manager.cc:194] Registered new tserver with Master: b1ea8e9e6ef545b7a0bb2ebe46b75413 (127.2.39.194:38443)
WARNING: no policy specified for oryx/127.2.39.195@KRBTEST.COM; defaulting to no policy
I20260926 06:25:29.254112 8307 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:39695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab.
Entry for principal oryx/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab.
I20260926 06:25:29.271880 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33879
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab
--principal=oryx/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:29.350899 8637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:29.351083 8637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:29.351105 8637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:29.352257 8637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:29.352299 8637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:29.352324 8637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:29.353736 8637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab
--principal=oryx/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33879
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:29.353914 8637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:29.354140 8637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:29.359059 8642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.359056 8645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:29.359143 8637 server_base.cc:1034] running on GCE node
W20260926 06:25:29.359059 8643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:29.359445 8637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:29.359660 8637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:29.360797 8637 hybrid_clock.cc:648] HybridClock initialized: now 1790403929360763 us; error 50 us; skew 500 ppm
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:29.363204 8637 init.cc:377] Logged in from keytab as oryx/127.2.39.195@KRBTEST.COM (short username oryx)
I20260926 06:25:29.363582 8637 webserver.cc:460] Webserver started at http://127.2.39.195:33093/ using document root <none> and password file <none>
I20260926 06:25:29.363812 8637 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:29.363880 8637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:29.364029 8637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:29.365000 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "461aaf9065df46d3a750517ec79b5a95"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "c4ec3b67d843d8a4cd49ff0e2815338a"
server_key_iv: "899d388491cc3de7934e649de6c024fd"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.365347 8637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "461aaf9065df46d3a750517ec79b5a95"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "c4ec3b67d843d8a4cd49ff0e2815338a"
server_key_iv: "899d388491cc3de7934e649de6c024fd"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.366729 8637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:25:29.367434 8652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.367610 8637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.367703 8637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "461aaf9065df46d3a750517ec79b5a95"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "c4ec3b67d843d8a4cd49ff0e2815338a"
server_key_iv: "899d388491cc3de7934e649de6c024fd"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.367787 8637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:29.374137 8637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:29.376276 8637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:29.376403 8637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:29.376631 8637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:29.376904 8637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:29.376936 8637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.376956 8637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:29.377002 8637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.382493 8637 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:42249
I20260926 06:25:29.382563 8765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:42249 every 8 connection(s)
I20260926 06:25:29.382866 8637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:29.386197 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 8637
I20260926 06:25:29.386300 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:29.386487 2207 external_mini_cluster.cc:1428] Setting key eec6114df269f28ee763d524023f19a0
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:29.391731 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.383574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:35199 (local address 127.2.39.254:33879)
0926 06:25:29.383620 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.383624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.383815 (+ 191us) server_negotiation.cc:407] Connection header received
0926 06:25:29.384242 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.384243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.384252 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.384304 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.385185 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.385755 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.387161 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.387437 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.387489 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.389402 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.389403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.389403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.389409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.390642 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.390983 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.390984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.390984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.391033 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.391242 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.391243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.391243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.391339 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.391390 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.391617 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.391671 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:25:29.391885 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.383558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.195:35199)
0926 06:25:29.383706 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.383737 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.384166 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.384366 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.384369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.384902 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.385089 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.385093 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.385829 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.385830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.387079 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.387081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.387506 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.387507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.387620 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.388026 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.388035 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.389326 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.390710 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.390715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.390717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.390920 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.391089 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.391089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.391090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.391185 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.391397 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.391400 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.391501 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.391589 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.391698 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260926 06:25:29.392381 8766 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33879
I20260926 06:25:29.392488 8766 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:29.392655 8766 heartbeater.cc:511] Master 127.2.39.254:33879 requested a full tablet report, sending...
I20260926 06:25:29.393024 8307 ts_manager.cc:194] Registered new tserver with Master: 461aaf9065df46d3a750517ec79b5a95 (127.2.39.195:42249)
I20260926 06:25:29.393389 8307 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:35199
I20260926 06:25:29.398137 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.39.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.253.keytab.
Entry for principal oryx/127.2.39.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.253.keytab.
I20260926 06:25:29.424392 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.253:44323
--webserver_interface=127.2.39.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.253.keytab
--principal=oryx/127.2.39.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:33879,127.2.39.253:44323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:29.503939 8771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:29.504130 8771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:29.504153 8771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:29.505268 8771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:29.505308 8771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:29.505322 8771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:29.505359 8771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:29.505388 8771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:29.506914 8771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:33879,127.2.39.253:44323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.253.keytab
--principal=oryx/127.2.39.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.253:44323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.39.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
W20260926 06:25:29.507030 8771 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.39.254:33879,127.2.39.253:44323'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 06:25:29.507165 8771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:29.507404 8771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:29.512094 8777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.512126 8776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.512127 8779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:29.512354 8771 server_base.cc:1034] running on GCE node
I20260926 06:25:29.512655 8771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:29.512969 8771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:29.514132 8771 hybrid_clock.cc:648] HybridClock initialized: now 1790403929514109 us; error 35 us; skew 500 ppm
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:29.516528 8771 init.cc:377] Logged in from keytab as oryx/127.2.39.253@KRBTEST.COM (short username oryx)
I20260926 06:25:29.516995 8771 webserver.cc:460] Webserver started at http://127.2.39.253:43229/ using document root <none> and password file <none>
I20260926 06:25:29.517172 8771 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:29.517212 8771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:29.517298 8771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:29.518332 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data/instance:
uuid: "72294669dbe241fbabfb60c8942016f6"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "e297752a4f4739d23dbf36d56ed72983"
server_key_iv: "63e9b0daf5b0b566e5c8452e3c4d3bce"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.518632 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal/instance:
uuid: "72294669dbe241fbabfb60c8942016f6"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "e297752a4f4739d23dbf36d56ed72983"
server_key_iv: "63e9b0daf5b0b566e5c8452e3c4d3bce"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.519915 8771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:25:29.520587 8786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.520776 8771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:29.520872 8771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal
uuid: "72294669dbe241fbabfb60c8942016f6"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "e297752a4f4739d23dbf36d56ed72983"
server_key_iv: "63e9b0daf5b0b566e5c8452e3c4d3bce"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.520959 8771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:29.535523 8771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:29.537451 8771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:29.537664 8771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:29.541558 8771 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.253:44323
I20260926 06:25:29.541613 8838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.253:44323 every 8 connection(s)
I20260926 06:25:29.541996 8771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
I20260926 06:25:29.543016 8839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:29.544477 8839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:25:29.548861 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 8771
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.253@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:29.553967 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.545166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.2.39.254:33879)
0926 06:25:29.545275 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.545277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.545499 (+ 222us) server_negotiation.cc:407] Connection header received
0926 06:25:29.546084 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.546086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.546099 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.546159 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.546778 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.547287 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.548308 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.548498 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.548594 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.551180 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.551182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.551182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.551196 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.552560 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.552978 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.552980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.552980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.553045 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.553293 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.553294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.553294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.553435 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.553507 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.553806 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.553880 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 06:25:29.554181 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.545089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:47874)
0926 06:25:29.545316 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.545408 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.545968 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.546233 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.546236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.546491 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.546674 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.546678 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.547421 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.547422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.548199 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.548201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.548611 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.548613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.548755 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.549311 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.549322 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.551083 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.552682 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.552688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.552690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.552903 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.553113 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.553114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.553114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.553226 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.553509 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.553515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.553682 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.553786 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.553926 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.39.253@KRBTEST.COM
I20260926 06:25:29.554796 8839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:25:29.557996 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.549401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.2.39.253:44323)
0926 06:25:29.549565 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.549592 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.549678 (+ 86us) server_negotiation.cc:407] Connection header received
0926 06:25:29.549708 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.549710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.549737 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.549817 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.550507 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.551268 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.552030 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.552316 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.552463 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.554955 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.554961 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.554963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.554976 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.556607 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.557005 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.557007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.557008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.557064 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.557273 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.557274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.557274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.557442 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.557617 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.557746 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.557855 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.253@KRBTEST.COM for oryx/127.2.39.253@KRBTEST.COM
I20260926 06:25:29.561818 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.555057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:44323 (local address 127.0.0.1:34990)
0926 06:25:29.555097 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.555107 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.555173 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.555645 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.555646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.555769 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.555809 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.555810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.556439 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.556440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.556897 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.556898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.557262 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.557263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.557327 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.557728 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.557732 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.558985 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.561065 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.561065 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.561066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.561203 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.561346 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.561346 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.561346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.561390 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.561548 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.561549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.561631 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.561690 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.561767 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260926 06:25:29.561985 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.555141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.2.39.253:44323)
0926 06:25:29.555516 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.555518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.555526 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:29.555546 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.555546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.555556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.555633 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.555894 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.556363 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.556986 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.557182 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.557262 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.559053 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.559054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.559054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.559059 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.560999 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.561254 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.561254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.561255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.561297 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.561425 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.561426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.561426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.561501 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.561542 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.561722 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.561809 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":355,"thread_start_us":86,"threads_started":1}
I20260926 06:25:29.563239 8839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: Bootstrap starting.
I20260926 06:25:29.563802 8839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:29.564108 8839 log.cc:824] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:29.564795 8839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: No bootstrap required, opened a new log
I20260926 06:25:29.566293 8839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } }
I20260926 06:25:29.566424 8839 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:29.566473 8839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72294669dbe241fbabfb60c8942016f6, State: Initialized, Role: FOLLOWER
I20260926 06:25:29.566608 8839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } }
I20260926 06:25:29.566890 8839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:29.566859 8846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } } }
I20260926 06:25:29.567000 8846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:25:29.568475 8857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:25:29.568550 8857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:25:29.570073 8771 master_runner.cc:428] Detected that this master 72294669dbe241fbabfb60c8942016f6 is joining an existing cluster
I20260926 06:25:29.570124 8771 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.39.253:44323
I20260926 06:25:29.577879 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.571136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47882 (local address 127.2.39.254:33879)
0926 06:25:29.571176 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.571178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.571333 (+ 155us) server_negotiation.cc:407] Connection header received
0926 06:25:29.571420 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.571421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.571438 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.571497 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.572022 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.572740 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.573521 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.573768 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.573846 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.574957 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.574959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.574959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.574967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.576660 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.577043 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.577044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.577045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.577097 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.577313 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.577314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.577314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.577439 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.577481 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.577726 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.577791 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:25:29.577895 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.571098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:47882)
0926 06:25:29.571273 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.571282 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.571362 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.571542 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.571543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.571797 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.571920 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.571922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.572822 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.572823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.573428 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.573430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.573908 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.573909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.573966 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.574428 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.574431 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.574867 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.576740 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.576742 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.576742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.576973 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.577169 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.577170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.577170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.577232 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.577539 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.577540 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.577604 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.577696 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.577791 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":38,"thread_start_us":87,"threads_started":1}
I20260926 06:25:29.578011 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.571166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35000 (local address 127.2.39.253:44323)
0926 06:25:29.571206 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.571207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.571384 (+ 177us) server_negotiation.cc:407] Connection header received
0926 06:25:29.571479 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.571479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.571487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.571526 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.571972 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.572725 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.573816 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.574141 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.574233 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.575030 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.575031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.575031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.575037 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.576775 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.577200 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.577201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.577201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.577239 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.577458 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.577460 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.577460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.577605 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.577671 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.577858 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.577932 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:25:29.578563 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.571195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:44323 (local address 127.0.0.1:35000)
0926 06:25:29.571332 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.571342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.571431 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.571544 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.571544 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.571790 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.571899 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.571900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.572804 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.572805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.573736 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.573739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.574235 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.574236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.574292 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.574590 (+ 298us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.574595 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.574963 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.576855 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.576856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.576857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.577110 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.577298 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.577299 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.577299 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.577369 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.577666 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.577667 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.577721 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.578362 (+ 641us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.578467 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":28,"thread_start_us":106,"threads_started":1}
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403928, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.254@KRBTEST.COM for oryx/127.2.39.253@KRBTEST.COM
I20260926 06:25:29.586892 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.579956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.2.39.253:44323)
0926 06:25:29.579991 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.579993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.580133 (+ 140us) server_negotiation.cc:407] Connection header received
0926 06:25:29.580223 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.580223 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.580230 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.580273 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.580731 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.581134 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.582243 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.582406 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.582456 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.584435 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.584436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.584436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.584442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.585764 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.586134 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.586135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.586136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.586193 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.586391 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.586392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.586393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.586485 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.586557 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.586733 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.586801 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:25:29.586967 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.579878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:44323 (local address 127.0.0.1:35006)
0926 06:25:29.580038 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.580083 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.580155 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.580278 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.580281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.580608 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.580661 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.580663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.581207 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.581208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.582150 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.582152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.582477 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.582478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.582558 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.582941 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.582946 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.584338 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.585855 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.585861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.585863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.586060 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.586257 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.586258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.586258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.586316 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.586562 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.586569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.586615 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.586708 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.586810 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260926 06:25:29.587598 8307 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.39.253:44323: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "72294669dbe241fbabfb60c8942016f6"
member_type: NON_VOTER
last_known_addr {
host: "127.2.39.253"
port: 44323
}
attrs {
promote: true
}
}
dest_uuid: "3e4964c5316c40d9b58b83994c8db608"
cas_config_opid_index: -1
I20260926 06:25:29.587878 8307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } }
I20260926 06:25:29.588335 8866 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 72294669dbe241fbabfb60c8942016f6 (127.2.39.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } } }
I20260926 06:25:29.588598 8813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:29.589327 8866 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.39.253:44323
I20260926 06:25:29.589337 8813 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 FOLLOWER]: Refusing update from remote peer 3e4964c5316c40d9b58b83994c8db608: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 06:25:29.589560 8771 master.cc:537] Master@127.2.39.253:44323 shutting down...
I20260926 06:25:29.589951 8866 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:29.589950 8868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } } }
I20260926 06:25:29.590576 8868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: This master's current role is: LEADER
W20260926 06:25:29.590827 8293 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 06:25:29.592036 8293 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 -> Peer 72294669dbe241fbabfb60c8942016f6 (127.2.39.253:44323): Couldn't send request to peer 72294669dbe241fbabfb60c8942016f6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 06:25:29.592640 8771 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:29.592787 8771 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:29.592871 8771 tablet_replica.cc:330] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: stopping tablet replica
I20260926 06:25:29.596467 8771 master.cc:552] Master@127.2.39.253:44323 shutdown complete.
I20260926 06:25:29.596541 8771 master_runner.cc:305] Clearing existing system tablet
I20260926 06:25:29.596679 8771 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:29.596725 8771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:29.597460 8771 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:29.604013 8872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.604117 8771 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:29.604163 8771 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260926 06:25:29.604187 8771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal
uuid: "72294669dbe241fbabfb60c8942016f6"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "e297752a4f4739d23dbf36d56ed72983"
server_key_iv: "63e9b0daf5b0b566e5c8452e3c4d3bce"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.604965 8771 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:29.606547 8771 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 06:25:29.606613 8771 log.cc:1196] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 06:25:29.606809 8771 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: Deleting consensus metadata
I20260926 06:25:29.606938 8771 master_runner.cc:315] Copying system tablet from 127.2.39.254:33879
I20260926 06:25:29.607610 8771 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: tablet copy: Beginning tablet copy session from remote peer at address 127.2.39.254:33879
I20260926 06:25:29.614647 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.607927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:47896)
0926 06:25:29.608172 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.608221 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.608340 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.608521 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.608522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.608719 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.608790 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.608792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.609827 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.609828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.610459 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.610461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.610945 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.610946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.610998 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.611736 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.611741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.612036 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.613705 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.613706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.613707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.613869 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.614070 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.614071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.614071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.614154 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.614414 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.614414 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.614449 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.614499 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.614573 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20260926 06:25:29.614684 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.608100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47896 (local address 127.2.39.254:33879)
0926 06:25:29.608141 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.608143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.608301 (+ 158us) server_negotiation.cc:407] Connection header received
0926 06:25:29.608432 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.608433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.608445 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.608503 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.608890 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.609733 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.610549 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.610858 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.610932 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.612122 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.612123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.612124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.612130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.613620 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.613940 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.613941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.613942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.614002 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.614225 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.614226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.614226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.614351 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.614418 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.614522 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.614594 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:25:29.614997 8327 tablet_copy_service.cc:140] P 3e4964c5316c40d9b58b83994c8db608: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 72294669dbe241fbabfb60c8942016f6 ({username='oryx', principal='oryx/127.2.39.253@KRBTEST.COM'} at 127.0.0.1:47896)
I20260926 06:25:29.615105 8327 tablet_copy_service.cc:161] P 3e4964c5316c40d9b58b83994c8db608: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 72294669dbe241fbabfb60c8942016f6 at {username='oryx', principal='oryx/127.2.39.253@KRBTEST.COM'} at 127.0.0.1:47896: session id = 72294669dbe241fbabfb60c8942016f6-00000000000000000000000000000000
I20260926 06:25:29.615896 8327 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 06:25:29.616482 8771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 06:25:29.617360 8771 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: tablet copy: Starting download of 0 data blocks...
I20260926 06:25:29.617548 8771 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: tablet copy: Starting download of 1 WAL segments...
I20260926 06:25:29.618561 8771 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 06:25:29.619597 8327 tablet_copy_service.cc:342] P 3e4964c5316c40d9b58b83994c8db608: Request end of tablet copy session 72294669dbe241fbabfb60c8942016f6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.39.253@KRBTEST.COM'} at 127.0.0.1:47896
I20260926 06:25:29.619653 8327 tablet_copy_service.cc:433] P 3e4964c5316c40d9b58b83994c8db608: ending tablet copy session 72294669dbe241fbabfb60c8942016f6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 72294669dbe241fbabfb60c8942016f6
I20260926 06:25:29.622179 8771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:29.623383 8879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:29.623448 8771 server_base.cc:1034] running on GCE node
W20260926 06:25:29.623416 8880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.623492 8882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:29.623775 8771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:29.623903 8771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:29.625021 8771 hybrid_clock.cc:648] HybridClock initialized: now 1790403929625008 us; error 29 us; skew 500 ppm
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:29.626317 8771 init.cc:377] Logged in from keytab as oryx/127.2.39.253@KRBTEST.COM (short username oryx)
I20260926 06:25:29.626609 8771 webserver.cc:460] Webserver started at http://127.2.39.253:34141/ using document root <none> and password file <none>
I20260926 06:25:29.626726 8771 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:29.626760 8771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:29.627354 8771 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:29.627861 8889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.628049 8771 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:29.628100 8771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal
uuid: "72294669dbe241fbabfb60c8942016f6"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "e297752a4f4739d23dbf36d56ed72983"
server_key_iv: "63e9b0daf5b0b566e5c8452e3c4d3bce"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.628159 8771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:29.650316 8771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:29.651976 8771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:29.655069 8771 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.253:44323
I20260926 06:25:29.655212 8941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.253:44323 every 8 connection(s)
I20260926 06:25:29.655421 8771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
I20260926 06:25:29.656138 8942 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:25:29.656543 8942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: Bootstrap starting.
I20260926 06:25:29.658622 8942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:25:29.658854 8942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: Bootstrap complete.
I20260926 06:25:29.659047 8942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } }
I20260926 06:25:29.659111 8942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 72294669dbe241fbabfb60c8942016f6, State: Initialized, Role: LEARNER
I20260926 06:25:29.659137 8942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } }
I20260926 06:25:29.659308 8942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:29.659334 8945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } } }
I20260926 06:25:29.659548 8945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: This master's current role is: LEARNER
I20260926 06:25:29.661250 8957 catalog_manager.cc:1259] Loaded cluster ID: 9b4400c3851c4f079ff38b0eaf864b50
I20260926 06:25:29.661386 8957 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: loading cluster ID for follower catalog manager: success
I20260926 06:25:29.662729 8957 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: acquiring CA information for follower catalog manager: success
I20260926 06:25:29.663033 8957 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:25:29.667045 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.659612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.2.39.253:44323)
0926 06:25:29.659757 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.659759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.659770 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:25:29.659794 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.659794 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.659804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.659884 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.660975 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.662027 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.662980 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.663278 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.663358 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.664413 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.664414 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.664415 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.664421 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.665981 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.666308 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.666309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.666309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.666372 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.666553 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.666554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.666554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.666677 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.666778 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.666855 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.666942 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.2.39.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.252.keytab.
Entry for principal oryx/127.2.39.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.252.keytab.
I20260926 06:25:29.691648 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.252:36137
--webserver_interface=127.2.39.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.252.keytab
--principal=oryx/127.2.39.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:33879,127.2.39.253:44323,127.2.39.252:36137
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:29.775514 8960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:29.775714 8960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:29.775738 8960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:29.776932 8960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:29.776973 8960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:29.776988 8960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:29.777001 8960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:29.777055 8960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:29.778573 8960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:33879,127.2.39.253:44323,127.2.39.252:36137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.252.keytab
--principal=oryx/127.2.39.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.252:36137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.39.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:29.778822 8960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:29.779048 8960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:29.783679 8965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.783728 8968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.783741 8966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:29.783957 8960 server_base.cc:1034] running on GCE node
I20260926 06:25:29.784284 8960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:29.784559 8960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:29.785691 8960 hybrid_clock.cc:648] HybridClock initialized: now 1790403929785670 us; error 33 us; skew 500 ppm
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:29.788100 8960 init.cc:377] Logged in from keytab as oryx/127.2.39.252@KRBTEST.COM (short username oryx)
I20260926 06:25:29.788565 8960 webserver.cc:460] Webserver started at http://127.2.39.252:43321/ using document root <none> and password file <none>
I20260926 06:25:29.788785 8960 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:29.788851 8960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:29.788996 8960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:29.790082 8960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data/instance:
uuid: "f9ab2d797b3f4673a62beedc83471a1b"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "7315ec5945b1df6158821817e0d06413"
server_key_iv: "6cc7174af0fbeeb737ae60f05e2c55a5"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.790431 8960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal/instance:
uuid: "f9ab2d797b3f4673a62beedc83471a1b"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "7315ec5945b1df6158821817e0d06413"
server_key_iv: "6cc7174af0fbeeb737ae60f05e2c55a5"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.791688 8960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:29.792414 8975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.792593 8960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:29.792688 8960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal
uuid: "f9ab2d797b3f4673a62beedc83471a1b"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "7315ec5945b1df6158821817e0d06413"
server_key_iv: "6cc7174af0fbeeb737ae60f05e2c55a5"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.792765 8960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:29.813728 8960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:29.816087 8960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:29.816236 8960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:29.821050 8960 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.252:36137
I20260926 06:25:29.821099 9027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.252:36137 every 8 connection(s)
I20260926 06:25:29.821429 8960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
I20260926 06:25:29.822505 9028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:29.824057 9028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:25:29.826004 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 8960
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.252@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.39.252@KRBTEST.COM
I20260926 06:25:29.834491 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.824894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.2.39.254:33879)
0926 06:25:29.824957 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.824959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.825019 (+ 60us) server_negotiation.cc:407] Connection header received
0926 06:25:29.825603 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.825604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.825615 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.825679 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.826375 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.827020 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.828413 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.828654 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.828739 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.831198 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.831199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.831200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.831208 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.832755 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.833395 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.833396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.833396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.833457 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.833769 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.833770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.833770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.833918 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.834022 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.834301 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.834399 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:25:29.834666 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.824706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:47912)
0926 06:25:29.824885 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.824948 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.825441 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.825766 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.825769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.826029 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.826264 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.826269 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.827126 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.827127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.828338 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.828340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.828747 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.828748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.828838 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.829311 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.829318 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.831110 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.832855 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.832862 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.832865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.833235 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.833535 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.833536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.833537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.833685 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.834015 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.834021 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.834158 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.834282 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.834429 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260926 06:25:29.835300 9028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:25:29.835414 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.826458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.2.39.252:36137)
0926 06:25:29.826633 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.826637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.826658 (+ 21us) server_negotiation.cc:407] Connection header received
0926 06:25:29.826679 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.826680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.826698 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.826773 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.827372 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.828492 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.829482 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.829956 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.830026 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.832447 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.832455 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.832455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.832461 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.834125 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.834492 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.834494 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.834495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.834579 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.834795 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.834796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.834796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.834942 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.835120 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.835255 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.835328 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.252@KRBTEST.COM for oryx/127.2.39.253@KRBTEST.COM
I20260926 06:25:29.841549 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.835571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:44323 (local address 127.0.0.1:35032)
0926 06:25:29.835612 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.835622 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.835657 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.835835 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.835836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.835987 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.836044 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.836045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.836646 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.836647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.837127 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.837128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.837447 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.837448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.837497 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.837839 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.837842 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.838994 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.840577 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.840577 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.840578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.840733 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.840943 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.840943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.840944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.840998 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.841247 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.841248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.841287 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.841372 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.841456 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 06:25:29.841554 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.835661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.2.39.253:44323)
0926 06:25:29.835707 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.835709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.835717 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:29.835740 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.835740 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.835751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.835809 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.836124 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.836579 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.837199 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.837385 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.837432 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.839084 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.839086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.839086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.839093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.840466 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.840815 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.840816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.840817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.840871 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.841073 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.841074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.841075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.841175 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.841243 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.841397 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.841470 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:25:29.841958 9028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.252@KRBTEST.COM for oryx/127.2.39.252@KRBTEST.COM
I20260926 06:25:29.847635 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.842182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36137 (local address 127.0.0.1:37688)
0926 06:25:29.842229 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.842239 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.842284 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.842403 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.842404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.842535 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.842556 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.842557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.843166 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.843167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.843624 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.843625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.843993 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.843994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.844041 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.844420 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.844425 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.845448 (+ 1023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.846847 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.846848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.846848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.846998 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.847173 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.847174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.847174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.847223 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.847438 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.847439 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.847469 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.847516 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.847583 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 06:25:29.847694 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.842207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.2.39.252:36137)
0926 06:25:29.842234 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.842236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.842271 (+ 35us) server_negotiation.cc:407] Connection header received
0926 06:25:29.842337 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.842337 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.842346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.842398 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.842632 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.843077 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.843710 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.843915 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.843993 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.845545 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.845546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.845546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.845552 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.846777 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.847072 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.847072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.847073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.847118 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.847261 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.847261 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.847261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.847341 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.847436 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.847533 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.847624 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":14}
I20260926 06:25:29.848881 9028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Bootstrap starting.
I20260926 06:25:29.849727 9028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:29.850163 9028 log.cc:824] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:29.850893 9028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: No bootstrap required, opened a new log
I20260926 06:25:29.852669 9028 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } }
I20260926 06:25:29.852790 9028 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:29.852815 9028 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ab2d797b3f4673a62beedc83471a1b, State: Initialized, Role: FOLLOWER
I20260926 06:25:29.852958 9028 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } }
I20260926 06:25:29.853276 9028 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:29.853227 9033 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } } }
I20260926 06:25:29.853338 9033 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:25:29.855023 9044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:25:29.855095 9044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:25:29.857404 8960 master_runner.cc:428] Detected that this master f9ab2d797b3f4673a62beedc83471a1b is joining an existing cluster
I20260926 06:25:29.857448 8960 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.39.252:36137
I20260926 06:25:29.865741 9051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.858388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:47922)
0926 06:25:29.858704 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.858712 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.858793 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.858981 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.858982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.859151 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.859206 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.859208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.860111 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.860111 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.861073 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.861075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.861617 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.861618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.861680 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.862184 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.862189 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.862531 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.864308 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.864309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.864309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.864485 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.865069 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.865070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.865070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.865124 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.865430 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.865431 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.865466 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.865527 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.865668 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":51,"thread_start_us":90,"threads_started":1}
I20260926 06:25:29.865810 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.858594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.2.39.254:33879)
0926 06:25:29.858638 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.858640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.858755 (+ 115us) server_negotiation.cc:407] Connection header received
0926 06:25:29.858867 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.858868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.858878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.858918 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.859289 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.860015 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.861172 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.861473 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.861509 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.862631 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.862633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.862633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.862641 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.864219 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.864914 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.864915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.864915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.864987 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.865194 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.865195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.865195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.865332 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.865410 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.865609 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.865713 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403928, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.254@KRBTEST.COM for oryx/127.2.39.252@KRBTEST.COM
I20260926 06:25:29.885435 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.868097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36137 (local address 127.0.0.1:37704)
0926 06:25:29.868174 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.868185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.868234 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.868603 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.868604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.868745 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.868778 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.868780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.869469 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.869470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.870123 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.870125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.870503 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.870504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.870581 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.871039 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.871044 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.872413 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.884313 (+ 11900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.884314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.884315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.884541 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.884764 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.884765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.884765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.884821 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.885124 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.885125 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.885168 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.885218 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.885325 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 06:25:29.885892 9053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.868290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.2.39.252:36137)
0926 06:25:29.868434 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.868436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.868443 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:29.868483 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.868484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.868514 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.868584 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.868855 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.869370 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.870201 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.870417 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.870501 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.872513 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.872514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.872515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.872521 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.884203 (+ 11682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.884631 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.884632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.884632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.884687 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.884914 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.884915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.884915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.885043 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.885644 (+ 601us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.885687 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.885771 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260926 06:25:29.886546 8307 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.39.252:36137: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b"
member_type: NON_VOTER
last_known_addr {
host: "127.2.39.252"
port: 36137
}
attrs {
promote: true
}
}
dest_uuid: "3e4964c5316c40d9b58b83994c8db608"
cas_config_opid_index: 5
I20260926 06:25:29.886807 8307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } }
I20260926 06:25:29.887003 9050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.858381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:44323 (local address 127.0.0.1:35044)
0926 06:25:29.858539 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.858550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.858610 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.858791 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.858792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.859028 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.859157 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.859159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.860116 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.860117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.861073 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.861075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.861615 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.861616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.861680 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.882393 (+ 20713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.882399 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.884008 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.885928 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.885929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.885929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.886135 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.886355 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.886356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.886356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.886415 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.886676 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.886677 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.886715 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.886784 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.886893 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":47,"thread_start_us":93,"threads_started":1}
I20260926 06:25:29.887034 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.858441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.2.39.253:44323)
0926 06:25:29.858536 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.858539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.858615 (+ 76us) server_negotiation.cc:407] Connection header received
0926 06:25:29.858669 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.858670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.858681 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.858734 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.859247 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.860022 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.861172 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.861464 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.861506 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.884123 (+ 22617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.884124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.884125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.884134 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.885837 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.886220 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.886221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.886222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.886280 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.886473 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.886473 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.886474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.886592 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.886650 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.886849 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.886936 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 06:25:29.887178 8866 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f9ab2d797b3f4673a62beedc83471a1b (127.2.39.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.887509 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 8369
I20260926 06:25:29.888118 8866 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.39.252:36137
I20260926 06:25:29.888196 8866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.888267 8866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:29.888499 8960 master.cc:537] Master@127.2.39.252:36137 shutting down...
I20260926 06:25:29.892582 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.887311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:44323 (local address 127.0.0.1:35052)
0926 06:25:29.887507 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.887518 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.887556 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.888176 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.888177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.888179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:29.888219 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.888221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.889148 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.889149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.891428 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.891431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.892308 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.892310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.892379 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.892403 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.892474 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.892514 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260926 06:25:29.892612 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.887386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.2.39.253:44323)
0926 06:25:29.888019 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.888021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.888028 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:29.888047 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.888048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.888070 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.888126 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:29.888285 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.889053 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.891534 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.892222 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.892297 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.892374 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.892480 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.892521 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612}
I20260926 06:25:29.892818 8916 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 LEARNER]: Refusing update from remote peer 3e4964c5316c40d9b58b83994c8db608: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 06:25:29.893043 8868 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 06:25:29.893667 8916 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f9ab2d797b3f4673a62beedc83471a1b (127.2.39.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.894985 9055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.895093 9055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: This master's current role is: LEARNER
I20260926 06:25:29.895263 8960 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:29.895344 8960 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:29.895390 8960 tablet_replica.cc:330] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: stopping tablet replica
I20260926 06:25:29.895387 8945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e4964c5316c40d9b58b83994c8db608. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: NON_VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: true } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.895501 8945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: This master's current role is: LEARNER
W20260926 06:25:29.895664 8294 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 -> Peer f9ab2d797b3f4673a62beedc83471a1b (127.2.39.252:36137): Couldn't send request to peer f9ab2d797b3f4673a62beedc83471a1b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 06:25:29.895689 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:46281
--local_ip_for_outbound_sockets=127.2.39.193
--tserver_master_addrs=127.2.39.254:33879
--webserver_port=40411
--webserver_interface=127.2.39.193
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab
--principal=oryx/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.39.254:33879,127.2.39.253:44323
--tserver_master_addrs=127.2.39.254:33879,127.2.39.253:44323,127.2.39.252:36137 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:25:29.896533 9031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.858796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.2.39.252:36137)
0926 06:25:29.858843 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.858845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.859051 (+ 206us) server_negotiation.cc:407] Connection header received
0926 06:25:29.859160 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.859162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.859171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.859201 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.859636 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.860313 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.861308 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.861608 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.861653 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.882740 (+ 21087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.882742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.882742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.882749 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.884751 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.885157 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.885159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.885159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.885209 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.885415 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.885416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.885416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.885529 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.896091 (+ 10562us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.896357 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.896440 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:25:29.897740 9052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.858882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36137 (local address 127.0.0.1:37698)
0926 06:25:29.858992 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.859002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.859073 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.859279 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.859280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.859489 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.859546 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.859548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.860434 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.860434 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.861210 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.861212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.861725 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.861726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.861793 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.882079 (+ 20286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.882084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.882602 (+ 518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.884837 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.884838 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.884838 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.885074 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.885280 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.885281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.885282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.885332 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.896159 (+ 10827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.896160 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.896250 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.897011 (+ 761us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.897654 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260926 06:25:29.898630 8868 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608: attempting to promote NON_VOTER 72294669dbe241fbabfb60c8942016f6 to VOTER
I20260926 06:25:29.898808 8868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } }
I20260926 06:25:29.899236 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.899131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36137 (local address 127.0.0.1:37716)
0926 06:25:29.899170 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.899208 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 06:25:29.899371 8916 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 LEARNER]: Refusing update from remote peer 3e4964c5316c40d9b58b83994c8db608: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 06:25:29.899506 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.899437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36137 (local address 127.0.0.1:37726)
0926 06:25:29.899467 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.899487 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 06:25:29.899566 8294 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 -> Peer f9ab2d797b3f4673a62beedc83471a1b (127.2.39.252:36137): Couldn't send request to peer f9ab2d797b3f4673a62beedc83471a1b. Status: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36137: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 06:25:29.899681 8867 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 06:25:29.900300 8866 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72294669dbe241fbabfb60c8942016f6 (127.2.39.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.900518 8916 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 72294669dbe241fbabfb60c8942016f6 (127.2.39.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.901142 8866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.901216 8866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:29.901270 8866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.901317 8866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:29.901501 8945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.901616 8945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:25:29.909770 8960 master.cc:552] Master@127.2.39.252:36137 shutdown complete.
I20260926 06:25:29.909821 8960 master_runner.cc:305] Clearing existing system tablet
I20260926 06:25:29.909962 8960 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:29.910013 8960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:29.910898 8960 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:29.911455 9058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.911609 8960 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:29.911659 8960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:29.911680 8960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal
uuid: "f9ab2d797b3f4673a62beedc83471a1b"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "7315ec5945b1df6158821817e0d06413"
server_key_iv: "6cc7174af0fbeeb737ae60f05e2c55a5"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.912465 8960 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:29.913910 8960 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 06:25:29.913976 8960 log.cc:1196] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 06:25:29.914184 8960 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Deleting consensus metadata
I20260926 06:25:29.914304 8960 master_runner.cc:315] Copying system tablet from 127.2.39.254:33879
I20260926 06:25:29.914870 8960 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: tablet copy: Beginning tablet copy session from remote peer at address 127.2.39.254:33879
I20260926 06:25:29.922566 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.915175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:47938)
0926 06:25:29.915329 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:29.915341 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:29.915415 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:29.915574 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:29.915574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:29.915754 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:29.915836 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.915838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.916760 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.916761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.917671 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:29.917673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.918138 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.918139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:29.918207 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.919193 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:29.919198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:29.919573 (+ 375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:29.921450 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.921451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.921452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.921686 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.921902 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:29.921911 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:29.921911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:29.921977 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:29.922258 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:29.922258 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:29.922301 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 06:25:29.922370 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:25:29.922466 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260926 06:25:29.922866 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:29.915230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.2.39.254:33879)
0926 06:25:29.915279 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:29.915281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:29.915423 (+ 142us) server_negotiation.cc:407] Connection header received
0926 06:25:29.915483 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:29.915484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:29.915497 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:29.915559 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:29.915926 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.916671 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.917760 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:29.918050 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:29.918133 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:29.919687 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:29.919689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:29.919689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:29.919697 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:29.921359 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.921764 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.921765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.921765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.921826 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:29.922054 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:29.922055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:29.922056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:29.922187 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:29.922257 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:29.922408 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 06:25:29.922483 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:25:29.923266 8327 tablet_copy_service.cc:140] P 3e4964c5316c40d9b58b83994c8db608: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f9ab2d797b3f4673a62beedc83471a1b ({username='oryx', principal='oryx/127.2.39.252@KRBTEST.COM'} at 127.0.0.1:47938)
I20260926 06:25:29.923319 8327 tablet_copy_service.cc:161] P 3e4964c5316c40d9b58b83994c8db608: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f9ab2d797b3f4673a62beedc83471a1b at {username='oryx', principal='oryx/127.2.39.252@KRBTEST.COM'} at 127.0.0.1:47938: session id = f9ab2d797b3f4673a62beedc83471a1b-00000000000000000000000000000000
I20260926 06:25:29.923870 8327 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 06:25:29.924382 8960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 06:25:29.925554 8960 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: tablet copy: Starting download of 0 data blocks...
I20260926 06:25:29.925746 8960 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: tablet copy: Starting download of 1 WAL segments...
I20260926 06:25:29.926403 8960 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 06:25:29.927201 8327 tablet_copy_service.cc:342] P 3e4964c5316c40d9b58b83994c8db608: Request end of tablet copy session f9ab2d797b3f4673a62beedc83471a1b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.39.252@KRBTEST.COM'} at 127.0.0.1:47938
I20260926 06:25:29.927253 8327 tablet_copy_service.cc:433] P 3e4964c5316c40d9b58b83994c8db608: ending tablet copy session f9ab2d797b3f4673a62beedc83471a1b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f9ab2d797b3f4673a62beedc83471a1b
I20260926 06:25:29.929683 8960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:29.931121 9065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.931133 9066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:29.934546 9068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:29.934676 8960 server_base.cc:1034] running on GCE node
I20260926 06:25:29.934839 8960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:29.934990 8960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:29.936116 8960 hybrid_clock.cc:648] HybridClock initialized: now 1790403929936101 us; error 42 us; skew 500 ppm
Sep 26 06:25:29 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403929, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:29.937542 8960 init.cc:377] Logged in from keytab as oryx/127.2.39.252@KRBTEST.COM (short username oryx)
I20260926 06:25:29.937814 8960 webserver.cc:460] Webserver started at http://127.2.39.252:32919/ using document root <none> and password file <none>
I20260926 06:25:29.937929 8960 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:29.937966 8960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:29.938755 8960 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:29.939337 9075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:29.939486 8960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:29.939536 8960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal
uuid: "f9ab2d797b3f4673a62beedc83471a1b"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "7315ec5945b1df6158821817e0d06413"
server_key_iv: "6cc7174af0fbeeb737ae60f05e2c55a5"
server_key_version: "encryptionkey@0"
I20260926 06:25:29.939587 8960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:29.950280 8960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:29.952082 8960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:29.955237 8960 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.252:36137
I20260926 06:25:29.955489 9127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.252:36137 every 8 connection(s)
I20260926 06:25:29.955734 8960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
I20260926 06:25:29.956657 9128 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:25:29.957146 9128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Bootstrap starting.
I20260926 06:25:29.960076 9128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:25:29.960304 9128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Bootstrap complete.
I20260926 06:25:29.960543 9128 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } }
I20260926 06:25:29.960639 9128 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f9ab2d797b3f4673a62beedc83471a1b, State: Initialized, Role: LEARNER
I20260926 06:25:29.960675 9128 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } }
I20260926 06:25:29.960855 9128 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:29.960837 9131 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:29.961083 9131 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: This master's current role is: LEARNER
I20260926 06:25:29.963346 9142 catalog_manager.cc:1259] Loaded cluster ID: 9b4400c3851c4f079ff38b0eaf864b50
I20260926 06:25:29.963434 9142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: loading cluster ID for follower catalog manager: success
I20260926 06:25:29.964593 9142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: acquiring CA information for follower catalog manager: success
I20260926 06:25:29.964939 9142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 06:25:29.993130 9056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:29.993325 9056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:29.993356 9056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:29.994547 9056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:29.994589 9056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:29.994616 9056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:29.996079 9056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab
--principal=oryx/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:46281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=40411
--tserver_master_addrs=127.2.39.254:33879,127.2.39.253:44323,127.2.39.252:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:29.996313 9056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:29.996600 9056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:30.002611 9147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:30.002599 9148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:30.002957 9150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:30.004191 9056 server_base.cc:1034] running on GCE node
I20260926 06:25:30.004412 9056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:30.004671 9056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:30.005837 9056 hybrid_clock.cc:648] HybridClock initialized: now 1790403930005815 us; error 40 us; skew 500 ppm
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:30.009416 9056 init.cc:377] Logged in from keytab as oryx/127.2.39.193@KRBTEST.COM (short username oryx)
I20260926 06:25:30.009994 9056 webserver.cc:460] Webserver started at http://127.2.39.193:40411/ using document root <none> and password file <none>
I20260926 06:25:30.010244 9056 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:30.010336 9056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:30.011916 9056 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:25:30.012647 9157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:30.012903 9056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:30.012997 9056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "57c3d1ab035f49e488da5c43d18427cc"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "71ee776d440bc71766bdcd3c88c89e70"
server_key_iv: "542b6e60c7a048cb22a544930c658d1b"
server_key_version: "encryptionkey@0"
I20260926 06:25:30.013437 9056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:30.024086 9056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:30.026145 9056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:30.026263 9056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:30.026470 9056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:30.026793 9056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:30.026831 9056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:30.026878 9056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:30.026908 9056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:30.033710 9056 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:46281
I20260926 06:25:30.033775 9270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:46281 every 8 connection(s)
I20260926 06:25:30.034158 9056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:30.034701 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 9056
I20260926 06:25:30.034873 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 8503
I20260926 06:25:30.040767 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:38443
--local_ip_for_outbound_sockets=127.2.39.194
--tserver_master_addrs=127.2.39.254:33879
--webserver_port=45131
--webserver_interface=127.2.39.194
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab
--principal=oryx/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.39.254:33879,127.2.39.253:44323
--tserver_master_addrs=127.2.39.254:33879,127.2.39.253:44323,127.2.39.252:36137 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.252@KRBTEST.COM
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.253@KRBTEST.COM
I20260926 06:25:30.048769 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.035208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:41399 (local address 127.2.39.254:33879)
0926 06:25:30.035255 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.035258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.036321 (+ 1063us) server_negotiation.cc:407] Connection header received
0926 06:25:30.037418 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.037420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.037432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.037501 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.038486 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.039262 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.041165 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.041457 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.041498 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.044984 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.044985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.044986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.044994 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.047008 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.047563 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.047564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.047564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.047628 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.047972 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.047973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.047974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.048118 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.048167 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.048567 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.048649 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:25:30.049353 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.034962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.193:41399)
0926 06:25:30.036188 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.036235 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.037283 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.037592 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.037597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.038058 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:30.038366 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.038372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.039380 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.039381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.041084 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.041086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.041605 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.041607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.041672 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.042627 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:30.042632 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:30.044861 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:30.047101 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.047108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.047111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.047452 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.047718 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.047719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.047719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.047877 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.048218 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:30.048224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:30.048385 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.048934 (+ 549us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.049119 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":47,"thread_start_us":109,"threads_started":1}
I20260926 06:25:30.049671 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.036322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:52109 (local address 127.2.39.252:36137)
0926 06:25:30.036484 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.036486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.038274 (+ 1788us) server_negotiation.cc:407] Connection header received
0926 06:25:30.038301 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.038302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.038314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.038371 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.038718 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.039688 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.040942 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.041245 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.041299 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.045963 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.045965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.045965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.045973 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.048013 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.048494 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.048496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.048496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.048557 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.048771 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.048772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.048772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.048934 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.049000 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.049413 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.049502 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":50,"threads_started":1}
I20260926 06:25:30.049876 9272 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33879
I20260926 06:25:30.049908 9276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.036240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36137 (local address 127.2.39.193:52109)
0926 06:25:30.038152 (+ 1912us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.038163 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.038220 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.038426 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.038427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.038585 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:30.038636 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.038637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.039772 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.039772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.040827 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.040830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.041357 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.041358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.041496 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.042390 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:30.042401 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:30.045873 (+ 3472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:30.048115 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.048116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.048117 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.048397 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.048632 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.048633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.048633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.048697 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.049028 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:30.049029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:30.049075 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.049146 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.049255 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1896,"mutex_wait_us":8,"thread_start_us":63,"threads_started":1}
I20260926 06:25:30.049975 9272 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:30.050174 9272 heartbeater.cc:511] Master 127.2.39.254:33879 requested a full tablet report, sending...
I20260926 06:25:30.050207 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.034962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:44323 (local address 127.2.39.193:57511)
0926 06:25:30.036356 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.036365 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.037294 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.037681 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.037682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.038056 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:30.038365 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.038371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.039482 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.039483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.040833 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.040836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.041407 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.041408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.041499 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.042704 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:30.042708 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:30.045493 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:30.048216 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.048217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.048217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.048509 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.048729 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.048730 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.048730 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.048790 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.049390 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:30.049391 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:30.049705 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.049997 (+ 292us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.050095 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"mutex_wait_us":95,"thread_start_us":87,"threads_started":1}
I20260926 06:25:30.050359 9271 heartbeater.cc:347] Connected to a master server at 127.2.39.252:36137
I20260926 06:25:30.050400 9271 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:30.050463 9271 heartbeater.cc:511] Master 127.2.39.252:36137 requested a full tablet report, sending...
I20260926 06:25:30.050734 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.035427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:57511 (local address 127.2.39.253:44323)
0926 06:25:30.035498 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.035500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.036421 (+ 921us) server_negotiation.cc:407] Connection header received
0926 06:25:30.037557 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.037559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.037592 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.037635 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.038470 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.039315 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.040981 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.041298 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.041355 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.045623 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.045624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.045624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.045633 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.048124 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.048592 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.048594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.048594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.048653 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.049129 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.049130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.049130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.049277 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.049359 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.050550 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.050638 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:25:30.050854 8307 ts_manager.cc:194] Re-registered known tserver with Master: 57c3d1ab035f49e488da5c43d18427cc (127.2.39.193:46281)
I20260926 06:25:30.050964 9092 ts_manager.cc:194] Registered new tserver with Master: 57c3d1ab035f49e488da5c43d18427cc (127.2.39.193:46281)
I20260926 06:25:30.051381 8307 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:41399
I20260926 06:25:30.051769 9273 heartbeater.cc:347] Connected to a master server at 127.2.39.253:44323
I20260926 06:25:30.051832 9273 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:30.051908 9273 heartbeater.cc:511] Master 127.2.39.253:44323 requested a full tablet report, sending...
I20260926 06:25:30.052292 8906 ts_manager.cc:194] Registered new tserver with Master: 57c3d1ab035f49e488da5c43d18427cc (127.2.39.193:46281)
W20260926 06:25:30.125188 9279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:30.125380 9279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:30.125402 9279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:30.126632 9279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:30.126677 9279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:30.126705 9279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:30.128224 9279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab
--principal=oryx/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:38443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=45131
--tserver_master_addrs=127.2.39.254:33879,127.2.39.253:44323,127.2.39.252:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:30.128460 9279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:30.128727 9279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:30.134052 9285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:30.134069 9284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:30.134105 9279 server_base.cc:1034] running on GCE node
W20260926 06:25:30.134070 9287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:30.134413 9279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:30.134680 9279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:30.135903 9279 hybrid_clock.cc:648] HybridClock initialized: now 1790403930135880 us; error 31 us; skew 500 ppm
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:30.138630 9279 init.cc:377] Logged in from keytab as oryx/127.2.39.194@KRBTEST.COM (short username oryx)
I20260926 06:25:30.139176 9279 webserver.cc:460] Webserver started at http://127.2.39.194:45131/ using document root <none> and password file <none>
I20260926 06:25:30.139430 9279 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:30.139518 9279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:30.141189 9279 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:30.141872 9294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:30.142062 9279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:30.142160 9279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "1f2184cfa81daad61100068d12d92da3"
server_key_iv: "ae352c28eaed95e359f6642cedd3a69c"
server_key_version: "encryptionkey@0"
I20260926 06:25:30.142536 9279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:30.162835 9279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:30.164716 9279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:30.164881 9279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:30.165117 9279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:30.165423 9279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:30.165475 9279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:30.165520 9279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:30.165547 9279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:30.171499 9279 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:38443
I20260926 06:25:30.171597 9407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:38443 every 8 connection(s)
I20260926 06:25:30.171924 9279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:30.176462 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 9279
I20260926 06:25:30.176635 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 8637
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.252@KRBTEST.COM
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.253@KRBTEST.COM
I20260926 06:25:30.183750 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.172622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:44461 (local address 127.2.39.252:36137)
0926 06:25:30.172677 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.172679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.172926 (+ 247us) server_negotiation.cc:407] Connection header received
0926 06:25:30.173558 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.173559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.173603 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.173665 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.174393 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.175045 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.176233 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.176565 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.176648 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.179411 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.179412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.179412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.179420 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.182154 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.182644 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.182645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.182646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.182702 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.182982 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.182983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.182983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.183121 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.183165 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.183559 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.183655 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:25:30.183846 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:42249
--local_ip_for_outbound_sockets=127.2.39.195
--tserver_master_addrs=127.2.39.254:33879
--webserver_port=33093
--webserver_interface=127.2.39.195
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab
--principal=oryx/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.39.254:33879,127.2.39.253:44323
--tserver_master_addrs=127.2.39.254:33879,127.2.39.253:44323,127.2.39.252:36137 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:25:30.184162 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.173039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:39451 (local address 127.2.39.253:44323)
0926 06:25:30.173086 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.173088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.173185 (+ 97us) server_negotiation.cc:407] Connection header received
0926 06:25:30.173561 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.173562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.173591 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.173626 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.174398 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.175185 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.177469 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.177779 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.177844 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.181288 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.181289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.181290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.181297 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.183027 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.183386 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.183387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.183387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.183441 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.183638 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.183639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.183639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.183759 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.183801 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.183993 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.184079 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:25:30.184307 9412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.172639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36137 (local address 127.2.39.194:44461)
0926 06:25:30.172794 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.172828 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.173443 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.173743 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.173754 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.174101 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:30.174289 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.174293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.175162 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.175163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.176155 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.176157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.176652 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.176653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.176823 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.177359 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:30.177370 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:30.179314 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:30.182232 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.182239 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.182242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.182546 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.182778 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.182779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.182780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.182906 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.183210 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:30.183216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:30.183386 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.183890 (+ 504us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.184071 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260926 06:25:30.184481 9413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.172979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:44323 (local address 127.2.39.194:39451)
0926 06:25:30.173131 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.173139 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.173466 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.174125 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.174126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.174261 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:30.174307 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.174309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.175281 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.175281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.176250 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.176252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.178344 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.178345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.178421 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.178821 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:30.178825 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:30.181177 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:30.183103 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.183103 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.183104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.183307 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.183511 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.183512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.183512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.183568 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.183846 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:30.183847 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:30.183886 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.184275 (+ 389us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.184370 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":54,"thread_start_us":68,"threads_started":1}
I20260926 06:25:30.184834 9410 heartbeater.cc:347] Connected to a master server at 127.2.39.253:44323
I20260926 06:25:30.184929 9410 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:30.185139 9410 heartbeater.cc:511] Master 127.2.39.253:44323 requested a full tablet report, sending...
I20260926 06:25:30.185176 9408 heartbeater.cc:347] Connected to a master server at 127.2.39.252:36137
I20260926 06:25:30.185256 9408 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:30.185330 9408 heartbeater.cc:511] Master 127.2.39.252:36137 requested a full tablet report, sending...
I20260926 06:25:30.185352 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.172634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:50393 (local address 127.2.39.254:33879)
0926 06:25:30.172726 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.172729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.173167 (+ 438us) server_negotiation.cc:407] Connection header received
0926 06:25:30.173587 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.173588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.173600 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.173632 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.174399 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.175181 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.177894 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.178232 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.178294 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.180673 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.180674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.180674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.180683 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.183211 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.183594 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.183595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.183596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.183658 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.184807 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.184809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.184809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.184947 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.185005 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.185191 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.185264 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 06:25:30.185626 8906 ts_manager.cc:194] Registered new tserver with Master: b1ea8e9e6ef545b7a0bb2ebe46b75413 (127.2.39.194:38443)
I20260926 06:25:30.185652 9092 ts_manager.cc:194] Registered new tserver with Master: b1ea8e9e6ef545b7a0bb2ebe46b75413 (127.2.39.194:38443)
I20260926 06:25:30.185700 9414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.172686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.194:50393)
0926 06:25:30.173098 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.173110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.173463 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.173758 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.173759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.174123 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:30.174290 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.174297 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.175281 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.175281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.176236 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.176238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.178583 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.178584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.178641 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.179282 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:30.179286 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:30.180570 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:30.183292 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.183293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.183293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.183510 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.183728 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.183729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.183729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.183788 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.185043 (+ 1255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:30.185043 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:30.185081 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.185461 (+ 380us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.185599 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":59,"thread_start_us":65,"threads_started":1}
I20260926 06:25:30.186029 9409 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33879
I20260926 06:25:30.186076 9409 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:30.186170 9409 heartbeater.cc:511] Master 127.2.39.254:33879 requested a full tablet report, sending...
I20260926 06:25:30.186400 8307 ts_manager.cc:194] Re-registered known tserver with Master: b1ea8e9e6ef545b7a0bb2ebe46b75413 (127.2.39.194:38443)
I20260926 06:25:30.186728 8307 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:50393
W20260926 06:25:30.265651 9415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:30.265866 9415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:30.265903 9415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:30.266999 9415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:30.267061 9415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:30.267102 9415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:30.268532 9415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab
--principal=oryx/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:42249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=33093
--tserver_master_addrs=127.2.39.254:33879,127.2.39.253:44323,127.2.39.252:36137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:30.268784 9415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:30.269008 9415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:30.273880 9421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:30.273936 9423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:30.274091 9415 server_base.cc:1034] running on GCE node
W20260926 06:25:30.274125 9420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:30.274374 9415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:30.274612 9415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:30.275786 9415 hybrid_clock.cc:648] HybridClock initialized: now 1790403930275790 us; error 51 us; skew 500 ppm
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:30.283097 9415 init.cc:377] Logged in from keytab as oryx/127.2.39.195@KRBTEST.COM (short username oryx)
I20260926 06:25:30.283564 9415 webserver.cc:460] Webserver started at http://127.2.39.195:33093/ using document root <none> and password file <none>
I20260926 06:25:30.283788 9415 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:30.283867 9415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:30.285210 9415 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:30.285883 9430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:30.286090 9415 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:30.286188 9415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "461aaf9065df46d3a750517ec79b5a95"
format_stamp: "Formatted at 2026-09-26 06:25:29 on dist-test-slave-fkbn"
server_key: "c4ec3b67d843d8a4cd49ff0e2815338a"
server_key_iv: "899d388491cc3de7934e649de6c024fd"
server_key_version: "encryptionkey@0"
I20260926 06:25:30.286532 9415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:30.312091 9415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:30.314160 9415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:30.314324 9415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:30.314563 9415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:30.314874 9415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:30.314929 9415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:30.314971 9415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:30.314999 9415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:30.320932 9415 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:42249
I20260926 06:25:30.320979 9543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:42249 every 8 connection(s)
I20260926 06:25:30.321306 9415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.253@KRBTEST.COM
Sep 26 06:25:30 dist-test-slave-fkbn krb5kdc[8262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403930, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.252@KRBTEST.COM
I20260926 06:25:30.330485 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 9415
I20260926 06:25:30.333171 8346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.322309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:37545 (local address 127.2.39.254:33879)
0926 06:25:30.322751 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.322754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.322764 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:30.323722 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.323724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.323737 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.323799 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.324431 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.324927 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.325960 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.326190 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.326275 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.329626 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.329628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.329629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.329638 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.331450 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.332007 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.332008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.332008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.332073 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.332364 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.332365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.332366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.332514 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.332565 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.332958 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.333058 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415}
I20260926 06:25:30.333737 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.322029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.195:37545)
0926 06:25:30.322200 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.322240 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.323610 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.323874 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.323878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.324147 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:30.324332 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.324335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.325031 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.325032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.325877 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.325879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.326284 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.326286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.326457 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.327046 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:30.327057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:30.329497 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:30.331543 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.331549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.331551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.331944 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.332135 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.332136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.332137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.332270 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.332618 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:30.332624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:30.332792 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.333278 (+ 486us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.333448 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260926 06:25:30.334185 9545 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33879
I20260926 06:25:30.334286 9545 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:30.334486 9545 heartbeater.cc:511] Master 127.2.39.254:33879 requested a full tablet report, sending...
I20260926 06:25:30.334818 8307 ts_manager.cc:194] Re-registered known tserver with Master: 461aaf9065df46d3a750517ec79b5a95 (127.2.39.195:42249)
I20260926 06:25:30.335296 8307 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:37545
I20260926 06:25:30.335500 8946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.322420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:41025 (local address 127.2.39.253:44323)
0926 06:25:30.322504 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.322506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.322777 (+ 271us) server_negotiation.cc:407] Connection header received
0926 06:25:30.323733 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.323734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.323746 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.323779 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.324442 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.325202 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.326378 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.326667 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.326748 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.329758 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.329759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.329760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.329768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.331707 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.334365 (+ 2658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.334366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.334366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.334434 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.334956 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.334957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.334957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.335087 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.335158 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.335335 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.335409 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 06:25:30.336275 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.322029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:44323 (local address 127.2.39.195:41025)
0926 06:25:30.322380 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.322392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.323631 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.323890 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.323890 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.324209 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:30.324334 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.324336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.325310 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.325310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.326246 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.326248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.326749 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.326750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.326818 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.327178 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:30.327184 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:30.329665 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:30.331809 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.331809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.331810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.334206 (+ 2396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.334567 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.334568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.334568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.334626 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.335192 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:30.335193 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:30.335231 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.335790 (+ 559us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.335977 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":122,"spinlock_wait_cycles":7424,"thread_start_us":96,"threads_started":1}
I20260926 06:25:30.337919 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.322321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:50639 (local address 127.2.39.252:36137)
0926 06:25:30.322474 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.322476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.322777 (+ 301us) server_negotiation.cc:407] Connection header received
0926 06:25:30.323733 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.323735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.323747 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.323779 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.324448 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.325204 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.326818 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.327131 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.327196 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.331950 (+ 4754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.331951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.331952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.331959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.333638 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.337009 (+ 3371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.337010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.337011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.337072 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.337299 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.337300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.337300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.337440 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.337484 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.337747 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.337824 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260926 06:25:30.339407 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.322030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36137 (local address 127.2.39.195:50639)
0926 06:25:30.322557 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.322564 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.323634 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.323875 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.323878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.324206 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:30.324334 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.324336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.325309 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.325310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.326246 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.326248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.327212 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.327212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.327287 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.327710 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:30.327716 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:30.331717 (+ 4001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:30.334694 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.334695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.334696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.335824 (+ 1128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.337160 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:30.337161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:30.337161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:30.337222 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:30.337550 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:30.337551 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:30.337622 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.339167 (+ 1545us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.339277 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":11,"thread_start_us":96,"threads_started":1}
I20260926 06:25:30.339763 9546 heartbeater.cc:347] Connected to a master server at 127.2.39.253:44323
I20260926 06:25:30.339825 9546 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:30.339895 9546 heartbeater.cc:511] Master 127.2.39.253:44323 requested a full tablet report, sending...
I20260926 06:25:30.339989 9544 heartbeater.cc:347] Connected to a master server at 127.2.39.252:36137
I20260926 06:25:30.340023 9544 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:30.340065 9544 heartbeater.cc:511] Master 127.2.39.252:36137 requested a full tablet report, sending...
I20260926 06:25:30.340211 8906 ts_manager.cc:194] Registered new tserver with Master: 461aaf9065df46d3a750517ec79b5a95 (127.2.39.195:42249)
I20260926 06:25:30.340319 9092 ts_manager.cc:194] Registered new tserver with Master: 461aaf9065df46d3a750517ec79b5a95 (127.2.39.195:42249)
I20260926 06:25:30.341042 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.332298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.2.39.252:36137)
0926 06:25:30.332674 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.332676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.332943 (+ 267us) server_negotiation.cc:407] Connection header received
0926 06:25:30.332965 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.332966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.332979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.333037 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.334422 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.335106 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.336179 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.336505 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.336561 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.337926 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.337927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.337927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.337933 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.339769 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.340138 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.340139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.340139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.340193 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.340391 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.340392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.340392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.340520 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.340628 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.340828 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.340926 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":69,"threads_started":1}
I20260926 06:25:30.341849 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.332074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.2.39.253:44323)
0926 06:25:30.333218 (+ 1144us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.333220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.333227 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:30.333244 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.333244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.333255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.333316 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.333982 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.334672 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.335914 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.336215 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.336290 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.337864 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.337865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.337866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.337871 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.339572 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.340846 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.340847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.340847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.340988 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.341190 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.341191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.341191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.341345 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.341434 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.341637 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.341717 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1112,"thread_start_us":42,"threads_started":1}
I20260926 06:25:30.343165 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.331760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.2.39.254:33879)
0926 06:25:30.332078 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.332080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.332401 (+ 321us) server_negotiation.cc:407] Connection header received
0926 06:25:30.332425 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.332426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.332439 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.332499 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.333672 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.336712 (+ 3040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.337809 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.338159 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.338219 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.339931 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.339932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.339933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.339939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.342070 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.342392 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.342393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.342394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.342446 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.342659 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.342660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.342661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.342770 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.342870 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.342972 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.343067 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
I20260926 06:25:30.344527 8307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:30.346256 8307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:30.353400 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.349721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:42249 (local address 127.0.0.1:50378)
0926 06:25:30.349921 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.349929 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.349968 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.350284 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.350285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.350286 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:30.350318 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.350319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.351204 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.351205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.352158 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.352160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.353166 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.353167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.353225 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.353230 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.353286 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.353328 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":57,"threads_started":1}
I20260926 06:25:30.353406 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.350008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.2.39.195:42249)
0926 06:25:30.350149 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.350154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.350163 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:25:30.350196 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.350197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.350212 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.350289 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:30.350372 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.351115 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.352481 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.353073 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.353155 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.353233 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.353300 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.353335 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260926 06:25:30.353636 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.349664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:38443 (local address 127.0.0.1:33782)
0926 06:25:30.349809 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.349819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.349856 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.350180 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.350181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.350183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:30.350222 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.350224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.351820 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.351821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.352502 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.352504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.353405 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.353406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.353468 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.353472 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.353502 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.353541 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260926 06:25:30.353943 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.349636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:46281 (local address 127.0.0.1:33624)
0926 06:25:30.349790 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.349802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.349864 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.350217 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.350218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.350220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:30.350261 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.350262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.351651 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.351652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.352683 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.352685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.353754 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.353755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.353816 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.353819 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.353851 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.353886 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260926 06:25:30.354274 9478 tablet_service.cc:1467] Processing CreateTablet for tablet 3a1131b554f54e3a85a356689c60627c (DEFAULT_TABLE table=test-table [id=200ec70ca1ba4c4f9f8782b28be41096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:30.354571 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.349747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.2.39.193:46281)
0926 06:25:30.350067 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.350070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.350078 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:30.350111 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.350113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.350128 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.350332 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:30.350400 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.351566 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.352760 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.353467 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.354048 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.354419 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.354448 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.354495 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":82,"threads_started":1}
I20260926 06:25:30.354676 9478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1131b554f54e3a85a356689c60627c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:30.355127 9205 tablet_service.cc:1467] Processing CreateTablet for tablet 3a1131b554f54e3a85a356689c60627c (DEFAULT_TABLE table=test-table [id=200ec70ca1ba4c4f9f8782b28be41096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:30.355422 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1131b554f54e3a85a356689c60627c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:30.357764 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.349861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33782 (local address 127.2.39.194:38443)
0926 06:25:30.350019 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.350025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.350035 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:30.350076 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.350078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.350095 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.350179 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:30.350397 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.351736 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.352593 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.353302 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.357463 (+ 4161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.357604 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.357631 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.357681 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260926 06:25:30.357969 9570 tablet_bootstrap.cc:492] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: Bootstrap starting.
I20260926 06:25:30.358139 9571 tablet_bootstrap.cc:492] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: Bootstrap starting.
I20260926 06:25:30.358407 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 3a1131b554f54e3a85a356689c60627c (DEFAULT_TABLE table=test-table [id=200ec70ca1ba4c4f9f8782b28be41096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:30.358610 9570 tablet_bootstrap.cc:654] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:30.358649 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a1131b554f54e3a85a356689c60627c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:30.358691 9571 tablet_bootstrap.cc:654] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:30.358945 9570 log.cc:824] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:30.358971 9571 log.cc:824] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:30.359727 9571 tablet_bootstrap.cc:492] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: No bootstrap required, opened a new log
I20260926 06:25:30.359772 9571 ts_tablet_manager.cc:1397] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:30.359966 9570 tablet_bootstrap.cc:492] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: No bootstrap required, opened a new log
I20260926 06:25:30.360023 9570 ts_tablet_manager.cc:1397] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:30.362118 9571 raft_consensus.cc:348] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:30.362268 9571 raft_consensus.cc:374] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:30.362308 9571 raft_consensus.cc:729] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c3d1ab035f49e488da5c43d18427cc, State: Initialized, Role: FOLLOWER
I20260926 06:25:30.362421 9571 consensus_queue.cc:260] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:30.362727 9574 tablet_bootstrap.cc:492] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: Bootstrap starting.
I20260926 06:25:30.362740 9272 heartbeater.cc:503] Master 127.2.39.254:33879 was elected leader, sending a full tablet report...
I20260926 06:25:30.363049 9570 raft_consensus.cc:348] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:30.363196 9570 raft_consensus.cc:374] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:30.363230 9570 raft_consensus.cc:729] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461aaf9065df46d3a750517ec79b5a95, State: Initialized, Role: FOLLOWER
I20260926 06:25:30.363340 9570 consensus_queue.cc:260] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:30.362700 9571 ts_tablet_manager.cc:1428] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:25:30.363584 9570 ts_tablet_manager.cc:1428] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 06:25:30.363667 9545 heartbeater.cc:503] Master 127.2.39.254:33879 was elected leader, sending a full tablet report...
I20260926 06:25:30.364624 9574 tablet_bootstrap.cc:654] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:30.364928 9574 log.cc:824] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:30.365696 9574 tablet_bootstrap.cc:492] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: No bootstrap required, opened a new log
I20260926 06:25:30.365772 9574 ts_tablet_manager.cc:1397] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 06:25:30.367043 9574 raft_consensus.cc:348] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:30.367182 9574 raft_consensus.cc:374] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:30.367225 9574 raft_consensus.cc:729] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ea8e9e6ef545b7a0bb2ebe46b75413, State: Initialized, Role: FOLLOWER
I20260926 06:25:30.367328 9574 consensus_queue.cc:260] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:30.367559 9574 ts_tablet_manager.cc:1428] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:25:30.367627 9409 heartbeater.cc:503] Master 127.2.39.254:33879 was elected leader, sending a full tablet report...
W20260926 06:25:30.423023 9411 tablet.cc:2369] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:25:30.469818 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.467240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36137 (local address 127.0.0.1:37746)
0926 06:25:30.467298 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.467310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.467356 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.467528 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.467529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.467530 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:30.467577 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.467578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.468186 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.468187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.468867 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.468868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.469532 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.469533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.469659 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.469663 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.469716 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.469758 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 06:25:30.469848 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.467328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.2.39.252:36137)
0926 06:25:30.467389 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.467391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.467402 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:25:30.467434 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.467435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.467454 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.467513 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:30.467632 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.468125 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.468970 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.469425 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.469501 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.469623 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.469731 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.469775 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:25:30.470285 9102 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260926 06:25:30.471240 9579 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e4964c5316c40d9b58b83994c8db608. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:30.471478 9579 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: This master's current role is: LEARNER
W20260926 06:25:30.537896 9274 tablet.cc:2369] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:25:30.543738 9575 raft_consensus.cc:482] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:30.543845 9575 raft_consensus.cc:504] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:30.544428 9575 leader_election.cc:288] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 461aaf9065df46d3a750517ec79b5a95 (127.2.39.195:42249), b1ea8e9e6ef545b7a0bb2ebe46b75413 (127.2.39.194:38443)
I20260926 06:25:30.547262 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.544641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:42249 (local address 127.2.39.193:32967)
0926 06:25:30.544700 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.544712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.544768 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.544926 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.544926 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.544928 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:30.545003 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.545005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.545682 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.545683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.546469 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.546470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.547000 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.547001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.547084 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.547087 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.547154 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.547195 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 06:25:30.547272 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.544717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:32967 (local address 127.2.39.195:42249)
0926 06:25:30.544799 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.544801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.544809 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:30.544845 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.544846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.544855 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.544922 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:30.545079 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.545545 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.546568 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.546910 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.547001 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.547010 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.547161 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.547199 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 06:25:30.547518 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.544734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:41589 (local address 127.2.39.194:38443)
0926 06:25:30.544795 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.544797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.544920 (+ 123us) server_negotiation.cc:407] Connection header received
0926 06:25:30.545021 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.545022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.545033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.545095 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:30.545252 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.545864 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.546764 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.547147 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.547230 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.547237 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.547414 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.547456 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:25:30.547520 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.544706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:38443 (local address 127.2.39.193:41589)
0926 06:25:30.544792 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:30.544831 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:30.544936 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:30.545111 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:30.545112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:30.545113 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:30.545185 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.545186 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.545934 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.545935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.546675 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:30.546677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.547227 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.547228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:30.547331 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.547334 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:30.547389 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:25:30.547439 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 06:25:30.547593 9498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1131b554f54e3a85a356689c60627c" candidate_uuid: "57c3d1ab035f49e488da5c43d18427cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "461aaf9065df46d3a750517ec79b5a95" is_pre_election: true
I20260926 06:25:30.547755 9498 raft_consensus.cc:2454] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c3d1ab035f49e488da5c43d18427cc in term 0.
I20260926 06:25:30.547874 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1131b554f54e3a85a356689c60627c" candidate_uuid: "57c3d1ab035f49e488da5c43d18427cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" is_pre_election: true
I20260926 06:25:30.548012 9160 leader_election.cc:302] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 461aaf9065df46d3a750517ec79b5a95, 57c3d1ab035f49e488da5c43d18427cc; no voters:
I20260926 06:25:30.548053 9362 raft_consensus.cc:2454] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c3d1ab035f49e488da5c43d18427cc in term 0.
I20260926 06:25:30.548130 9575 raft_consensus.cc:2781] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:30.548210 9575 raft_consensus.cc:482] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:30.548236 9575 raft_consensus.cc:3037] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:30.549366 9575 raft_consensus.cc:504] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:30.549536 9575 leader_election.cc:288] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [CANDIDATE]: Term 1 election: Requested vote from peers 461aaf9065df46d3a750517ec79b5a95 (127.2.39.195:42249), b1ea8e9e6ef545b7a0bb2ebe46b75413 (127.2.39.194:38443)
I20260926 06:25:30.549712 9498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1131b554f54e3a85a356689c60627c" candidate_uuid: "57c3d1ab035f49e488da5c43d18427cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "461aaf9065df46d3a750517ec79b5a95"
I20260926 06:25:30.549719 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a1131b554f54e3a85a356689c60627c" candidate_uuid: "57c3d1ab035f49e488da5c43d18427cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413"
I20260926 06:25:30.549789 9498 raft_consensus.cc:3037] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:30.549806 9362 raft_consensus.cc:3037] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:30.550511 9498 raft_consensus.cc:2454] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c3d1ab035f49e488da5c43d18427cc in term 1.
I20260926 06:25:30.550633 9362 raft_consensus.cc:2454] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c3d1ab035f49e488da5c43d18427cc in term 1.
I20260926 06:25:30.550663 9160 leader_election.cc:302] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 461aaf9065df46d3a750517ec79b5a95, 57c3d1ab035f49e488da5c43d18427cc; no voters:
I20260926 06:25:30.550740 9575 raft_consensus.cc:2781] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:30.550865 9575 raft_consensus.cc:686] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 1 LEADER]: Becoming Leader. State: Replica: 57c3d1ab035f49e488da5c43d18427cc, State: Running, Role: LEADER
I20260926 06:25:30.550977 9575 consensus_queue.cc:237] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:30.551745 8307 catalog_manager.cc:5495] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 57c3d1ab035f49e488da5c43d18427cc (127.2.39.193). New cstate: current_term: 1 leader_uuid: "57c3d1ab035f49e488da5c43d18427cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:30.552362 8868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: NON_VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: true } } }
I20260926 06:25:30.552459 8868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:30.552961 9587 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608: attempting to promote NON_VOTER f9ab2d797b3f4673a62beedc83471a1b to VOTER
I20260926 06:25:30.553102 9587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } }
I20260926 06:25:30.553722 9102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 1 LEARNER]: Refusing update from remote peer 3e4964c5316c40d9b58b83994c8db608: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 06:25:30.553725 8916 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 FOLLOWER]: Refusing update from remote peer 3e4964c5316c40d9b58b83994c8db608: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 06:25:30.553930 9587 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260926 06:25:30.554082 9589 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 06:25:30.554760 9587 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f9ab2d797b3f4673a62beedc83471a1b (127.2.39.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:30.554947 8916 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f9ab2d797b3f4673a62beedc83471a1b (127.2.39.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:30.554903 9102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f9ab2d797b3f4673a62beedc83471a1b (127.2.39.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:30.555732 9590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:30.555819 9590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:25:30.555842 8868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:30.555922 8868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:30.555888 9579 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:30.555972 9579 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:25:30.555996 8868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:30.556052 8868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:30.556118 9587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e4964c5316c40d9b58b83994c8db608" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:30.556169 9587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608 [sys.catalog]: This master's current role is: LEADER
W20260926 06:25:30.572077 9547 tablet.cc:2369] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:25:30.578930 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.576600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.2.39.193:46281)
0926 06:25:30.576666 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.576668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.576676 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:30.576711 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.576711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.576729 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.576789 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:30.576900 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.577396 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.578181 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.578371 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.578446 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.578565 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:30.578679 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:30.578738 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.578781 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.578800 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:25:30.580791 9498 raft_consensus.cc:1261] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95 [term 1 FOLLOWER]: Refusing update from remote peer 57c3d1ab035f49e488da5c43d18427cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:30.580789 9362 raft_consensus.cc:1261] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 1 FOLLOWER]: Refusing update from remote peer 57c3d1ab035f49e488da5c43d18427cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:30.581138 9575 consensus_queue.cc:1035] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:30.581307 9584 consensus_queue.cc:1035] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:30.584760 9594 mvcc.cc:205] Tried to move back new op lower bound from 7333494499655983104 to 7333494499537342464. Current Snapshot: MvccSnapshot[applied={T|T < 7333494499655983104}]
I20260926 06:25:30.584808 9593 mvcc.cc:205] Tried to move back new op lower bound from 7333494499655983104 to 7333494499537342464. Current Snapshot: MvccSnapshot[applied={T|T < 7333494499655983104}]
I20260926 06:25:30.585659 9592 mvcc.cc:205] Tried to move back new op lower bound from 7333494499655983104 to 7333494499537342464. Current Snapshot: MvccSnapshot[applied={T|T < 7333494499655983104 or (T in {7333494499655983104})}]
I20260926 06:25:30.586390 8307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:30.586474 8307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:30.588008 8307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e4964c5316c40d9b58b83994c8db608: Sending DeleteTablet for 3 replicas of tablet 3a1131b554f54e3a85a356689c60627c
I20260926 06:25:30.588332 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a1131b554f54e3a85a356689c60627c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:30 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:33782
I20260926 06:25:30.588332 9478 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a1131b554f54e3a85a356689c60627c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:30 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:50378
I20260926 06:25:30.588554 9602 tablet_replica.cc:330] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: stopping tablet replica
I20260926 06:25:30.588600 9205 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a1131b554f54e3a85a356689c60627c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:30 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:33624
I20260926 06:25:30.588654 9602 raft_consensus.cc:2229] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:30.588758 9602 raft_consensus.cc:2258] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:30.588922 9603 tablet_replica.cc:330] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: stopping tablet replica
I20260926 06:25:30.589017 9603 raft_consensus.cc:2229] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:30.589118 9603 raft_consensus.cc:2258] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:30.589385 9602 ts_tablet_manager.cc:1905] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:30.589589 9603 ts_tablet_manager.cc:1905] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:30.589720 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 8275
I20260926 06:25:30.591776 9602 ts_tablet_manager.cc:1918] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:30.591907 9602 log.cc:1196] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/3a1131b554f54e3a85a356689c60627c
I20260926 06:25:30.592017 9603 ts_tablet_manager.cc:1918] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:30.592101 9603 log.cc:1196] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/3a1131b554f54e3a85a356689c60627c
I20260926 06:25:30.592373 9603 ts_tablet_manager.cc:1939] T 3a1131b554f54e3a85a356689c60627c P b1ea8e9e6ef545b7a0bb2ebe46b75413: Deleting consensus metadata
I20260926 06:25:30.592399 9602 ts_tablet_manager.cc:1939] T 3a1131b554f54e3a85a356689c60627c P 461aaf9065df46d3a750517ec79b5a95: Deleting consensus metadata
I20260926 06:25:30.593945 9604 tablet_replica.cc:330] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: stopping tablet replica
I20260926 06:25:30.594096 9604 raft_consensus.cc:2229] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:30.594235 9604 raft_consensus.cc:2258] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 06:25:30.596105 9434 connection.cc:535] server connection from 127.0.0.1:50378 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:25:30.596477 9604 ts_tablet_manager.cc:1905] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 06:25:30.596494 9298 connection.cc:535] server connection from 127.0.0.1:33782 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:25:30.598650 9604 ts_tablet_manager.cc:1918] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:30.598722 9604 log.cc:1196] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/3a1131b554f54e3a85a356689c60627c
I20260926 06:25:30.598987 9604 ts_tablet_manager.cc:1939] T 3a1131b554f54e3a85a356689c60627c P 57c3d1ab035f49e488da5c43d18427cc: Deleting consensus metadata
W20260926 06:25:30.599287 9160 connection.cc:441] server connection from 127.0.0.1:33624 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33624 (request call id 1) could send its response
I20260926 06:25:30.602254 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.593837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.2.39.252:36137)
0926 06:25:30.593880 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.593893 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.593902 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:25:30.593924 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.593924 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.593934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.593996 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.594729 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.595504 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.596889 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.597182 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.597224 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.599312 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.599314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.599314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.599323 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.600928 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.601361 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.601362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.601362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.601418 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.601679 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.601680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.601681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.601811 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.601890 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.602086 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.602162 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:25:30.607728 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.599510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.2.39.253:44323)
0926 06:25:30.599548 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.599550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.599937 (+ 387us) server_negotiation.cc:407] Connection header received
0926 06:25:30.599981 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.599982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.599995 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.600054 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.600675 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.601434 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.602570 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.602884 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.602933 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.604337 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.604338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.604339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.604346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.605975 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.606636 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.606637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.606637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.606697 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.606912 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.606913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.606913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.607055 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.607160 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.607550 (+ 390us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.607642 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:25:30.608357 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:30.600080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37758 (local address 127.2.39.252:36137)
0926 06:25:30.600117 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:30.600119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:30.601147 (+ 1028us) server_negotiation.cc:407] Connection header received
0926 06:25:30.601237 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:30.601237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:30.601247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:30.601276 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:30.601911 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.602596 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.603653 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:30.603854 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:30.603934 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:30.605390 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:30.605391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:30.605391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:30.605398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:30.607015 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.607351 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.607352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.607353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.607406 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:30.607651 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:30.607652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:30.607652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:30.607774 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:30.607843 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:30.608211 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 06:25:30.608282 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:25:31.585004 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:31.584726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.193:39493)
0926 06:25:31.584851 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:31.584940 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
W20260926 06:25:31.585305 9272 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:33879 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
I20260926 06:25:31.585311 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:31.584938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.194:59595)
0926 06:25:31.585189 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:31.585264 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":49,"threads_started":1}
W20260926 06:25:31.585753 9409 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:33879 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
I20260926 06:25:31.586012 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:31.584809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.195:55985)
0926 06:25:31.584996 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:31.585954 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":109,"threads_started":1}
W20260926 06:25:31.586230 9545 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:33879 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
I20260926 06:25:32.088006 9615 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e4964c5316c40d9b58b83994c8db608)
I20260926 06:25:32.088133 9615 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } }
I20260926 06:25:32.088406 9616 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e4964c5316c40d9b58b83994c8db608)
I20260926 06:25:32.088436 9615 leader_election.cc:288] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e4964c5316c40d9b58b83994c8db608 (127.2.39.254:33879), 72294669dbe241fbabfb60c8942016f6 (127.2.39.253:44323)
I20260926 06:25:32.088625 9616 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } }
I20260926 06:25:32.088744 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.088517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:48368)
0926 06:25:32.088649 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.088700 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
W20260926 06:25:32.088873 9076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
I20260926 06:25:32.089221 9616 leader_election.cc:288] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e4964c5316c40d9b58b83994c8db608 (127.2.39.254:33879), f9ab2d797b3f4673a62beedc83471a1b (127.2.39.252:36137)
I20260926 06:25:32.089254 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.089146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:48394)
0926 06:25:32.089197 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.089230 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 06:25:32.089322 9076 leader_election.cc:334] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e4964c5316c40d9b58b83994c8db608 (127.2.39.254:33879): Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
I20260926 06:25:32.089396 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.089118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:48384)
0926 06:25:32.089311 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.089361 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":46,"threads_started":1}
W20260926 06:25:32.089497 8890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
I20260926 06:25:32.090126 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.090021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:48406)
0926 06:25:32.090074 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.090097 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 06:25:32.090196 8890 leader_election.cc:334] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e4964c5316c40d9b58b83994c8db608 (127.2.39.254:33879): Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
I20260926 06:25:32.091786 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.088578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:44323 (local address 127.0.0.1:35076)
0926 06:25:32.088713 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.088724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:32.088790 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:32.089041 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:32.089042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:32.089044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:32.089141 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:32.089143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.090128 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.090129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:32.090950 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:32.090952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.091531 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.091532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:32.091601 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.091604 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:32.091679 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:25:32.091721 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":39,"threads_started":1}
I20260926 06:25:32.091781 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.088657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.2.39.253:44323)
0926 06:25:32.088740 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:32.088742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:32.088781 (+ 39us) server_negotiation.cc:407] Connection header received
0926 06:25:32.088892 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:32.088893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:32.088908 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:32.088982 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:32.089246 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.090059 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.091081 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.091448 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.091541 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.091553 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:32.091686 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 06:25:32.091721 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":41,"threads_started":1}
I20260926 06:25:32.092023 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9ab2d797b3f4673a62beedc83471a1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72294669dbe241fbabfb60c8942016f6" is_pre_election: true
I20260926 06:25:32.092129 8916 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9ab2d797b3f4673a62beedc83471a1b in term 1.
I20260926 06:25:32.092310 9078 leader_election.cc:302] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72294669dbe241fbabfb60c8942016f6, f9ab2d797b3f4673a62beedc83471a1b; no voters: 3e4964c5316c40d9b58b83994c8db608
I20260926 06:25:32.092434 9615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 06:25:32.092506 9615 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3e4964c5316c40d9b58b83994c8db608)
I20260926 06:25:32.092546 9615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:25:32.092703 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.089306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36137 (local address 127.0.0.1:37770)
0926 06:25:32.089474 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.089484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:32.089542 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:32.089846 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:32.089847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:32.089849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:32.089887 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:32.089888 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.090733 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.090735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:32.091477 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:32.091479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.092474 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.092475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:32.092531 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.092534 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:32.092596 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:25:32.092638 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":49,"threads_started":1}
I20260926 06:25:32.092703 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.089611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.2.39.252:36137)
0926 06:25:32.089743 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:32.089745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:32.089752 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:32.089771 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:32.089772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:32.089780 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:32.089840 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:32.090017 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.090646 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.091582 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.092354 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.092432 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.092443 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:32.092615 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 06:25:32.092650 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260926 06:25:32.092895 9102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "72294669dbe241fbabfb60c8942016f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f9ab2d797b3f4673a62beedc83471a1b" is_pre_election: true
I20260926 06:25:32.093406 9615 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } }
I20260926 06:25:32.093525 9102 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72294669dbe241fbabfb60c8942016f6 in current term 2: Already voted for candidate f9ab2d797b3f4673a62beedc83471a1b in this term.
I20260926 06:25:32.093552 9615 leader_election.cc:288] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [CANDIDATE]: Term 2 election: Requested vote from peers 3e4964c5316c40d9b58b83994c8db608 (127.2.39.254:33879), 72294669dbe241fbabfb60c8942016f6 (127.2.39.253:44323)
I20260926 06:25:32.093700 8916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9ab2d797b3f4673a62beedc83471a1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "72294669dbe241fbabfb60c8942016f6"
I20260926 06:25:32.093735 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.093638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:48410)
0926 06:25:32.093687 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.093712 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 06:25:32.093767 8916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:25:32.093773 8893 leader_election.cc:302] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72294669dbe241fbabfb60c8942016f6; no voters: 3e4964c5316c40d9b58b83994c8db608, f9ab2d797b3f4673a62beedc83471a1b
I20260926 06:25:32.094071 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.093995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:48416)
0926 06:25:32.094030 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.094052 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 06:25:32.094156 9076 leader_election.cc:334] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3e4964c5316c40d9b58b83994c8db608 (127.2.39.254:33879): Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
I20260926 06:25:32.094512 8916 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9ab2d797b3f4673a62beedc83471a1b in term 2.
I20260926 06:25:32.094630 9616 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:25:32.094671 9078 leader_election.cc:302] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72294669dbe241fbabfb60c8942016f6, f9ab2d797b3f4673a62beedc83471a1b; no voters: 3e4964c5316c40d9b58b83994c8db608
I20260926 06:25:32.094753 9615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:25:32.094885 9615 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [term 2 LEADER]: Becoming Leader. State: Replica: f9ab2d797b3f4673a62beedc83471a1b, State: Running, Role: LEADER
I20260926 06:25:32.094965 9615 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } }
I20260926 06:25:32.095317 9625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9ab2d797b3f4673a62beedc83471a1b. Latest consensus state: current_term: 2 leader_uuid: "f9ab2d797b3f4673a62beedc83471a1b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:32.095391 9625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:32.095552 9627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:32.095961 9627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:32.096057 9627 catalog_manager.cc:1259] Loaded cluster ID: 9b4400c3851c4f079ff38b0eaf864b50
I20260926 06:25:32.096091 9627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:32.096275 9627 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:32.096382 9627 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Loaded TSK: 0
I20260926 06:25:32.096546 9627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:25:32.142447 9091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:32.144045 9091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:32.145129 8916 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [term 2 FOLLOWER]: Refusing update from remote peer f9ab2d797b3f4673a62beedc83471a1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 06:25:32.145326 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.144992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:48428)
0926 06:25:32.145262 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.145296 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251}
I20260926 06:25:32.145363 9625 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 06:25:32.145751 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.145641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.0.0.1:48434)
0926 06:25:32.145695 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.145725 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 06:25:32.145859 9076 consensus_peers.cc:487] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b -> Peer 3e4964c5316c40d9b58b83994c8db608 (127.2.39.254:33879): Couldn't send request to peer 3e4964c5316c40d9b58b83994c8db608. Status: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 06:25:32.146039 9616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9ab2d797b3f4673a62beedc83471a1b. Latest consensus state: current_term: 2 leader_uuid: "f9ab2d797b3f4673a62beedc83471a1b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:32.146152 9616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:25:32.146692 9625 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f9ab2d797b3f4673a62beedc83471a1b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:32.146773 9625 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:32.146842 9616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f9ab2d797b3f4673a62beedc83471a1b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3e4964c5316c40d9b58b83994c8db608" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33879 } } peers { permanent_uuid: "72294669dbe241fbabfb60c8942016f6" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 44323 } attrs { promote: false } } peers { permanent_uuid: "f9ab2d797b3f4673a62beedc83471a1b" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36137 } attrs { promote: false } } }
I20260926 06:25:32.146919 9616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72294669dbe241fbabfb60c8942016f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:25:32.151551 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.148482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:38443 (local address 127.0.0.1:33784)
0926 06:25:32.148677 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.148687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:32.148731 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:32.148928 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:32.148928 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:32.148930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:32.148974 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:32.148975 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.149889 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.149890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:32.150680 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:32.150682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.151344 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.151345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:32.151413 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.151419 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:25:32.151455 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 06:25:32.151493 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":74}
I20260926 06:25:32.151866 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.148574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.2.39.194:38443)
0926 06:25:32.148663 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:32.148665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:32.148719 (+ 54us) server_negotiation.cc:407] Connection header received
0926 06:25:32.148823 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:32.148824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:32.148836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:32.149041 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:32.149081 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.149794 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.150792 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.151233 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.151689 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.151707 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:32.151732 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 06:25:32.151779 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":47,"threads_started":1}
I20260926 06:25:32.152077 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.148461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:42249 (local address 127.0.0.1:50384)
0926 06:25:32.148576 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.148588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:32.148642 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:32.148823 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:32.148824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:32.148828 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:32.148904 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:32.148906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.149905 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.149905 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:32.151118 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:32.151119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.151864 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.151865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:32.151932 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.151935 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:32.151963 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 06:25:32.152003 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":58}
I20260926 06:25:32.152086 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.148555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:46281 (local address 127.0.0.1:33632)
0926 06:25:32.148794 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.148803 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:32.148859 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:32.148986 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:32.148987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:32.148989 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:32.149025 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:32.149027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.149898 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.149899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:32.151116 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:32.151118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.151875 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.151875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:32.151926 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.151930 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:32.151978 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:25:32.152006 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":58,"thread_start_us":87,"threads_started":1}
I20260926 06:25:32.152222 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.148775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.2.39.193:46281)
0926 06:25:32.148873 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:32.148875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:32.148883 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:32.148910 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:32.148911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:32.148922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:32.148984 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:32.149095 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.149819 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.151215 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.151767 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.151836 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.151849 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:32.152114 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 06:25:32.152164 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":50,"threads_started":1}
I20260926 06:25:32.152216 9342 tablet_service.cc:1467] Processing CreateTablet for tablet 4d63578af4b74199bc9fd93e910dbd78 (DEFAULT_TABLE table=test-table [id=ed47d9a47dbb47a0873f44299c046460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:32.152338 9342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d63578af4b74199bc9fd93e910dbd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:32.153358 9637 tablet_bootstrap.cc:492] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413: Bootstrap starting.
I20260926 06:25:32.153487 9205 tablet_service.cc:1467] Processing CreateTablet for tablet 4d63578af4b74199bc9fd93e910dbd78 (DEFAULT_TABLE table=test-table [id=ed47d9a47dbb47a0873f44299c046460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:32.153620 9205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d63578af4b74199bc9fd93e910dbd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:32.153627 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.148514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50384 (local address 127.2.39.195:42249)
0926 06:25:32.148671 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:32.148673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:32.148683 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:32.148707 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:32.148708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:32.148720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:32.148976 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:32.149015 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.149808 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.151215 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.151778 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.153446 (+ 1668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.153459 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:32.153477 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 06:25:32.153520 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":53,"threads_started":1}
I20260926 06:25:32.154173 9637 tablet_bootstrap.cc:654] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:32.154889 9637 tablet_bootstrap.cc:492] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413: No bootstrap required, opened a new log
I20260926 06:25:32.154950 9637 ts_tablet_manager.cc:1397] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:32.154901 9639 tablet_bootstrap.cc:492] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc: Bootstrap starting.
I20260926 06:25:32.155084 9478 tablet_service.cc:1467] Processing CreateTablet for tablet 4d63578af4b74199bc9fd93e910dbd78 (DEFAULT_TABLE table=test-table [id=ed47d9a47dbb47a0873f44299c046460]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:32.155138 9637 raft_consensus.cc:348] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:32.155222 9478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d63578af4b74199bc9fd93e910dbd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:32.155231 9637 raft_consensus.cc:374] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:32.155256 9637 raft_consensus.cc:729] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1ea8e9e6ef545b7a0bb2ebe46b75413, State: Initialized, Role: FOLLOWER
I20260926 06:25:32.155323 9637 consensus_queue.cc:260] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:32.155479 9639 tablet_bootstrap.cc:654] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:32.155499 9637 ts_tablet_manager.cc:1428] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:32.156126 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.155798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.194:42769)
0926 06:25:32.155951 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.156082 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":112,"threads_started":1}
I20260926 06:25:32.156432 9639 tablet_bootstrap.cc:492] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc: No bootstrap required, opened a new log
I20260926 06:25:32.156505 9639 ts_tablet_manager.cc:1397] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:32.156629 9639 raft_consensus.cc:348] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:32.156672 9643 tablet_bootstrap.cc:492] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95: Bootstrap starting.
I20260926 06:25:32.156708 9639 raft_consensus.cc:374] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:32.156735 9639 raft_consensus.cc:729] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c3d1ab035f49e488da5c43d18427cc, State: Initialized, Role: FOLLOWER
I20260926 06:25:32.156804 9639 consensus_queue.cc:260] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:32.156824 9408 heartbeater.cc:503] Master 127.2.39.252:36137 was elected leader, sending a full tablet report...
I20260926 06:25:32.156947 9639 ts_tablet_manager.cc:1428] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:32.157073 9643 tablet_bootstrap.cc:654] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:32.157552 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.157051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.194:34757)
0926 06:25:32.157484 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.157518 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412}
I20260926 06:25:32.157635 9643 tablet_bootstrap.cc:492] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95: No bootstrap required, opened a new log
I20260926 06:25:32.157684 9643 ts_tablet_manager.cc:1397] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:32.157758 9271 heartbeater.cc:503] Master 127.2.39.252:36137 was elected leader, sending a full tablet report...
I20260926 06:25:32.157842 9643 raft_consensus.cc:348] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:32.157922 9643 raft_consensus.cc:374] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:32.157950 9643 raft_consensus.cc:729] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461aaf9065df46d3a750517ec79b5a95, State: Initialized, Role: FOLLOWER
I20260926 06:25:32.157960 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.157767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.193:51305)
0926 06:25:32.157891 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.157923 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260926 06:25:32.157991 9643 consensus_queue.cc:260] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:32.158146 9643 ts_tablet_manager.cc:1428] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:32.158352 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.158149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.193:37343)
0926 06:25:32.158293 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.158321 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260926 06:25:32.158618 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.158378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.195:41579)
0926 06:25:32.158526 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.158569 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":102,"threads_started":1}
I20260926 06:25:32.158890 9544 heartbeater.cc:503] Master 127.2.39.252:36137 was elected leader, sending a full tablet report...
I20260926 06:25:32.159260 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.158830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.195:58395)
0926 06:25:32.159188 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.159228 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
I20260926 06:25:32.212441 9644 raft_consensus.cc:482] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:32.212540 9644 raft_consensus.cc:504] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:32.212726 9644 leader_election.cc:288] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 461aaf9065df46d3a750517ec79b5a95 (127.2.39.195:42249), b1ea8e9e6ef545b7a0bb2ebe46b75413 (127.2.39.194:38443)
I20260926 06:25:32.212949 9498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d63578af4b74199bc9fd93e910dbd78" candidate_uuid: "57c3d1ab035f49e488da5c43d18427cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "461aaf9065df46d3a750517ec79b5a95" is_pre_election: true
I20260926 06:25:32.212973 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d63578af4b74199bc9fd93e910dbd78" candidate_uuid: "57c3d1ab035f49e488da5c43d18427cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" is_pre_election: true
I20260926 06:25:32.213050 9498 raft_consensus.cc:2454] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c3d1ab035f49e488da5c43d18427cc in term 0.
I20260926 06:25:32.213073 9362 raft_consensus.cc:2454] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57c3d1ab035f49e488da5c43d18427cc in term 0.
I20260926 06:25:32.213184 9160 leader_election.cc:302] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 461aaf9065df46d3a750517ec79b5a95, 57c3d1ab035f49e488da5c43d18427cc; no voters:
I20260926 06:25:32.213282 9644 raft_consensus.cc:2781] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:32.213408 9644 raft_consensus.cc:482] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:32.213449 9644 raft_consensus.cc:3037] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:32.214190 9644 raft_consensus.cc:504] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:32.214330 9644 leader_election.cc:288] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [CANDIDATE]: Term 1 election: Requested vote from peers 461aaf9065df46d3a750517ec79b5a95 (127.2.39.195:42249), b1ea8e9e6ef545b7a0bb2ebe46b75413 (127.2.39.194:38443)
I20260926 06:25:32.214468 9498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d63578af4b74199bc9fd93e910dbd78" candidate_uuid: "57c3d1ab035f49e488da5c43d18427cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "461aaf9065df46d3a750517ec79b5a95"
I20260926 06:25:32.214478 9362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d63578af4b74199bc9fd93e910dbd78" candidate_uuid: "57c3d1ab035f49e488da5c43d18427cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413"
I20260926 06:25:32.214552 9498 raft_consensus.cc:3037] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:32.214570 9362 raft_consensus.cc:3037] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:32.215116 9498 raft_consensus.cc:2454] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c3d1ab035f49e488da5c43d18427cc in term 1.
I20260926 06:25:32.215224 9362 raft_consensus.cc:2454] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57c3d1ab035f49e488da5c43d18427cc in term 1.
I20260926 06:25:32.215324 9160 leader_election.cc:302] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 461aaf9065df46d3a750517ec79b5a95, 57c3d1ab035f49e488da5c43d18427cc; no voters:
I20260926 06:25:32.215417 9644 raft_consensus.cc:2781] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:32.215559 9644 raft_consensus.cc:686] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 1 LEADER]: Becoming Leader. State: Replica: 57c3d1ab035f49e488da5c43d18427cc, State: Running, Role: LEADER
I20260926 06:25:32.215628 9644 consensus_queue.cc:237] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } } peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } }
I20260926 06:25:32.215924 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.215806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.193:37277)
0926 06:25:32.215850 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.215888 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 06:25:32.216223 9091 catalog_manager.cc:5495] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 57c3d1ab035f49e488da5c43d18427cc (127.2.39.193). New cstate: current_term: 1 leader_uuid: "57c3d1ab035f49e488da5c43d18427cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57c3d1ab035f49e488da5c43d18427cc" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 46281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:32.224547 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.222526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.2.39.193:46281)
0926 06:25:32.222583 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:32.222585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:32.222592 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:32.222654 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:32.222654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:32.222663 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:32.222706 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:32.222817 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.223276 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.223972 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:32.224165 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:32.224224 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:32.224339 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:32.224369 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:32.224411 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:32.224457 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 06:25:32.224472 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:25:32.225274 9362 raft_consensus.cc:1261] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 1 FOLLOWER]: Refusing update from remote peer 57c3d1ab035f49e488da5c43d18427cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:32.225296 9498 raft_consensus.cc:1261] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95 [term 1 FOLLOWER]: Refusing update from remote peer 57c3d1ab035f49e488da5c43d18427cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:32.225467 9644 consensus_queue.cc:1035] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "461aaf9065df46d3a750517ec79b5a95" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:32.225548 9650 consensus_queue.cc:1035] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1ea8e9e6ef545b7a0bb2ebe46b75413" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:32.226243 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.226105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.195:45629)
0926 06:25:32.226177 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.226212 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 06:25:32.226788 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.226694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.194:43017)
0926 06:25:32.226734 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.226757 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 06:25:32.227370 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.227279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.193:56041)
0926 06:25:32.227312 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.227339 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 06:25:32.227705 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.227623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.195:40571)
0926 06:25:32.227658 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.227682 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 06:25:32.228027 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.227947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.193:44277)
0926 06:25:32.227981 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.228004 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 06:25:32.228770 9091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:32.228863 9091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:32.228931 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.228840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33879 (local address 127.2.39.194:59323)
0926 06:25:32.228887 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.228911 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 06:25:32.230057 9091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9ab2d797b3f4673a62beedc83471a1b: Sending DeleteTablet for 3 replicas of tablet 4d63578af4b74199bc9fd93e910dbd78
I20260926 06:25:32.230291 9478 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d63578af4b74199bc9fd93e910dbd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:32 UTC) from {username='oryx', principal='oryx/127.2.39.252@KRBTEST.COM'} at 127.0.0.1:50384
I20260926 06:25:32.230342 9205 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d63578af4b74199bc9fd93e910dbd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:32 UTC) from {username='oryx', principal='oryx/127.2.39.252@KRBTEST.COM'} at 127.0.0.1:33632
I20260926 06:25:32.230414 9342 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d63578af4b74199bc9fd93e910dbd78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:32 UTC) from {username='oryx', principal='oryx/127.2.39.252@KRBTEST.COM'} at 127.0.0.1:33784
I20260926 06:25:32.230479 9661 tablet_replica.cc:330] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95: stopping tablet replica
I20260926 06:25:32.230541 9661 raft_consensus.cc:2229] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:32.230579 9661 raft_consensus.cc:2258] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:32.230674 9662 tablet_replica.cc:330] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc: stopping tablet replica
I20260926 06:25:32.230741 9663 tablet_replica.cc:330] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413: stopping tablet replica
I20260926 06:25:32.230788 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 9056
I20260926 06:25:32.230800 9663 raft_consensus.cc:2229] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:32.230852 9663 raft_consensus.cc:2258] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:32.230872 9661 ts_tablet_manager.cc:1905] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:32.230770 9662 raft_consensus.cc:2229] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:32.231007 9662 raft_consensus.cc:2258] T 4d63578af4b74199bc9fd93e910dbd78 P 57c3d1ab035f49e488da5c43d18427cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:32.231151 9663 ts_tablet_manager.cc:1905] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:32.232450 9663 ts_tablet_manager.cc:1918] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:32.232537 9663 log.cc:1196] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/4d63578af4b74199bc9fd93e910dbd78
I20260926 06:25:32.232697 9661 ts_tablet_manager.cc:1918] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:32.232761 9661 log.cc:1196] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/4d63578af4b74199bc9fd93e910dbd78
I20260926 06:25:32.232806 9663 ts_tablet_manager.cc:1939] T 4d63578af4b74199bc9fd93e910dbd78 P b1ea8e9e6ef545b7a0bb2ebe46b75413: Deleting consensus metadata
I20260926 06:25:32.233011 9661 ts_tablet_manager.cc:1939] T 4d63578af4b74199bc9fd93e910dbd78 P 461aaf9065df46d3a750517ec79b5a95: Deleting consensus metadata
I20260926 06:25:32.233119 9077 catalog_manager.cc:4841] TS b1ea8e9e6ef545b7a0bb2ebe46b75413 (127.2.39.194:38443): tablet 4d63578af4b74199bc9fd93e910dbd78 (table test-table [id=ed47d9a47dbb47a0873f44299c046460]) successfully deleted
I20260926 06:25:32.233283 9078 catalog_manager.cc:4841] TS 461aaf9065df46d3a750517ec79b5a95 (127.2.39.195:42249): tablet 4d63578af4b74199bc9fd93e910dbd78 (table test-table [id=ed47d9a47dbb47a0873f44299c046460]) successfully deleted
I20260926 06:25:32.238147 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:32.238027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:46281 (local address 127.0.0.1:33648)
0926 06:25:32.238088 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:32.238113 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:46281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 06:25:32.238193 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 9279
W20260926 06:25:32.238315 9078 catalog_manager.cc:4584] TS 57c3d1ab035f49e488da5c43d18427cc (127.2.39.193:46281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d63578af4b74199bc9fd93e910dbd78: Network error: Client connection negotiation failed: client connection to 127.2.39.193:46281: connect: Connection refused (error 111)
I20260926 06:25:32.243899 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 9415
I20260926 06:25:32.248621 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 8771
I20260926 06:25:32.253187 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 8960
2026-09-26T06:25:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4735 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:32 dist-test-slave-fkbn krb5kdc[9666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:32 dist-test-slave-fkbn krb5kdc[9666](info): set up 2 sockets
Sep 26 06:25:32 dist-test-slave-fkbn krb5kdc[9666](info): commencing operation
krb5kdc: starting...
W20260926 06:25:33.286060 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:33 dist-test-slave-fkbn krb5kdc[9666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.254.keytab.
Entry for principal oryx/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.254.keytab.
I20260926 06:25:33.404616 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38243
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.254.keytab
--principal=oryx/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:38243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:33.486644 9679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:33.486891 9679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:33.486930 9679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:33.488071 9679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:33.488145 9679 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 06:25:33.488173 9679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:33.488206 9679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:33.488241 9679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:33.488268 9679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:33.489805 9679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:38243
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.254.keytab
--principal=oryx/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:33.490084 9679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:33.490362 9679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:33.495261 9684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:33.495307 9679 server_base.cc:1034] running on GCE node
W20260926 06:25:33.495275 9687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:33.495296 9685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:33.495781 9679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:33.496040 9679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:33.497189 9679 hybrid_clock.cc:648] HybridClock initialized: now 1790403933497174 us; error 29 us; skew 500 ppm
Sep 26 06:25:33 dist-test-slave-fkbn krb5kdc[9666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403933, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:33.499728 9679 init.cc:377] Logged in from keytab as oryx/127.2.39.254@KRBTEST.COM (short username oryx)
I20260926 06:25:33.500175 9679 webserver.cc:460] Webserver started at http://127.2.39.254:35963/ using document root <none> and password file <none>
I20260926 06:25:33.500372 9679 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:33.500415 9679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:33.500501 9679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:33.501493 9679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "1f673aa0e2e2450fb7f34bda46c45961"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "9d064b3741bd1ecde82a703c97002443"
server_key_iv: "4e49a0445de2b9e143fc9c71b4ecea8e"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.501868 9679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "1f673aa0e2e2450fb7f34bda46c45961"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "9d064b3741bd1ecde82a703c97002443"
server_key_iv: "4e49a0445de2b9e143fc9c71b4ecea8e"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.503073 9679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:33.503737 9694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:33.503912 9679 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:33.503985 9679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "1f673aa0e2e2450fb7f34bda46c45961"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "9d064b3741bd1ecde82a703c97002443"
server_key_iv: "4e49a0445de2b9e143fc9c71b4ecea8e"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.504037 9679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:33.512241 9679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:33.513990 9679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:33.514094 9679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:33.518802 9679 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:38243
I20260926 06:25:33.518864 9756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:38243 every 8 connection(s)
I20260926 06:25:33.519268 9679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:33.519865 9757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:33.522063 9757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961: Bootstrap starting.
I20260926 06:25:33.522588 9757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:33.522816 9757 log.cc:824] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:33.523443 9757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961: No bootstrap required, opened a new log
I20260926 06:25:33.525341 9757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f673aa0e2e2450fb7f34bda46c45961" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38243 } }
I20260926 06:25:33.525470 9757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:33.525491 9757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f673aa0e2e2450fb7f34bda46c45961, State: Initialized, Role: FOLLOWER
I20260926 06:25:33.525652 9757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f673aa0e2e2450fb7f34bda46c45961" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38243 } }
I20260926 06:25:33.525732 9757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:33.525754 9757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:33.525781 9757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:33.526257 9757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f673aa0e2e2450fb7f34bda46c45961" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38243 } }
I20260926 06:25:33.526337 9757 leader_election.cc:302] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f673aa0e2e2450fb7f34bda46c45961; no voters:
I20260926 06:25:33.526454 9757 leader_election.cc:288] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:33.526536 9760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:33.526713 9760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [term 1 LEADER]: Becoming Leader. State: Replica: 1f673aa0e2e2450fb7f34bda46c45961, State: Running, Role: LEADER
I20260926 06:25:33.526721 9757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:33.526813 9760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f673aa0e2e2450fb7f34bda46c45961" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38243 } }
I20260926 06:25:33.527189 9761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f673aa0e2e2450fb7f34bda46c45961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f673aa0e2e2450fb7f34bda46c45961" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38243 } } }
I20260926 06:25:33.527194 9762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f673aa0e2e2450fb7f34bda46c45961. Latest consensus state: current_term: 1 leader_uuid: "1f673aa0e2e2450fb7f34bda46c45961" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f673aa0e2e2450fb7f34bda46c45961" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38243 } } }
I20260926 06:25:33.527329 9762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:33.527524 9769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:33.527539 9761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:33.528059 9769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:33.529525 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 9679
I20260926 06:25:33.529655 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:33.529891 2207 external_mini_cluster.cc:1428] Setting key b72c611d6b9734e7c2005a16bd2a0e69
I20260926 06:25:33.530114 9769 catalog_manager.cc:1347] Generated new cluster ID: d74b929048f440fb9b63f06ba9dbb7af
I20260926 06:25:33.530159 9769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:33.533994 9769 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:33.534557 9769 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:33.543972 9769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961: Generated new TSK 0
I20260926 06:25:33.544153 9769 catalog_manager.cc:1514] Initializing in-progress tserver states...
Sep 26 06:25:33 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:33.549284 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:33.530348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.2.39.254:38243)
0926 06:25:33.537749 (+ 7401us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:33.537759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:33.537821 (+ 62us) server_negotiation.cc:407] Connection header received
0926 06:25:33.538016 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:33.538023 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:33.538090 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:33.538239 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:33.539009 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.540425 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.541603 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.542215 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.542316 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:33.545134 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:33.545138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:33.545138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:33.545148 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:33.547544 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:33.547990 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:33.547994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:33.547994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:33.548105 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:33.548323 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:33.548324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:33.548324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:33.548624 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:33.548752 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:33.548882 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 06:25:33.548994 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":7355,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab.
Entry for principal oryx/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab.
I20260926 06:25:33.576030 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38243
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab
--principal=oryx/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:33.664101 9783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:33.664300 9783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:33.664324 9783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:33.665397 9783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:33.665433 9783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:33.665454 9783 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:25:33.665472 9783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:33.666893 9783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab
--principal=oryx/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.39.254:38243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:33.667073 9783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:33.667240 9783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:33.672267 9789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:33.672263 9788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:33.672264 9791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:33.672657 9783 server_base.cc:1034] running on GCE node
I20260926 06:25:33.672883 9783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:33.673126 9783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:33.674264 9783 hybrid_clock.cc:648] HybridClock initialized: now 1790403933674249 us; error 27 us; skew 500 ppm
Sep 26 06:25:33 dist-test-slave-fkbn krb5kdc[9666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403933, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:33.676697 9783 init.cc:377] Logged in from keytab as oryx/127.2.39.193@KRBTEST.COM (short username oryx)
I20260926 06:25:33.677065 9783 webserver.cc:460] Webserver started at http://127.2.39.193:41377/ using document root <none> and password file <none>
I20260926 06:25:33.677253 9783 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:33.677291 9783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:33.677377 9783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:33.678418 9783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "048edfaedf2e45869146fcf867329c48"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "b92fdfcd5f3d250f79326a19cabda0d3"
server_key_iv: "727f0060eac43a2af7952386459c0916"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.678712 9783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "048edfaedf2e45869146fcf867329c48"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "b92fdfcd5f3d250f79326a19cabda0d3"
server_key_iv: "727f0060eac43a2af7952386459c0916"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.679905 9783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:25:33.680639 9798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:33.680814 9783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:33.680922 9783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "048edfaedf2e45869146fcf867329c48"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "b92fdfcd5f3d250f79326a19cabda0d3"
server_key_iv: "727f0060eac43a2af7952386459c0916"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.680997 9783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:33.690963 9783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:33.692965 9783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:33.693126 9783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:33.693679 9783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:33.693738 9783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:33.693778 9783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:33.693806 9783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:33.700784 9783 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:44087
I20260926 06:25:33.700892 9913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:44087 every 8 connection(s)
I20260926 06:25:33.701220 9783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:25:33 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403933, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:33.703171 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 9783
I20260926 06:25:33.703311 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:33.703545 2207 external_mini_cluster.cc:1428] Setting key 9305f5e775170f2553184033e0978af9
I20260926 06:25:33.705368 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:33.694121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:57961 (local address 127.2.39.254:38243)
0926 06:25:33.694198 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:33.694201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:33.694447 (+ 246us) server_negotiation.cc:407] Connection header received
0926 06:25:33.695204 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:33.695205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:33.695215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:33.695270 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:33.696246 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.697085 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.698756 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.699074 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.699153 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:33.702159 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:33.702160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:33.702161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:33.702168 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:33.703858 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:33.704302 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:33.704303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:33.704304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:33.704360 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:33.704615 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:33.704616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:33.704616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:33.704751 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:33.704817 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:33.705128 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 06:25:33.705225 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:25:33.705559 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:33.694107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38243 (local address 127.2.39.193:57961)
0926 06:25:33.694299 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:33.694347 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:33.695070 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:33.695353 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:33.695359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:33.695779 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:33.696123 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:33.696130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.697190 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.697191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:33.698659 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:33.698662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.699203 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.699205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:33.699380 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:33.700109 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:33.700120 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:33.702047 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:33.703952 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:33.703959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:33.703961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:33.704228 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:33.704433 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:33.704434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:33.704434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:33.704547 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:33.704817 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:33.704824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:33.704958 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 06:25:33.705086 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 06:25:33.705286 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260926 06:25:33.706153 9914 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38243
I20260926 06:25:33.706346 9914 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:33.706673 9914 heartbeater.cc:511] Master 127.2.39.254:38243 requested a full tablet report, sending...
I20260926 06:25:33.707340 9710 ts_manager.cc:194] Registered new tserver with Master: 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087)
I20260926 06:25:33.708102 9710 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:57961
WARNING: no policy specified for oryx/127.2.39.194@KRBTEST.COM; defaulting to no policy
I20260926 06:25:33.713379 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:33.707278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:41249 (local address 127.2.39.254:38243)
0926 06:25:33.707317 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:33.707319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:33.707656 (+ 337us) server_negotiation.cc:407] Connection header received
0926 06:25:33.707678 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:33.707679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:33.707689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:33.707740 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:33.708111 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.708681 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.709409 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.709676 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.709765 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:33.710673 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:33.710674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:33.710674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:33.710681 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:33.712056 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:33.712526 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:33.712527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:33.712528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:33.712586 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:33.712818 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:33.712819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:33.712820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:33.712953 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:33.713036 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:33.713209 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 06:25:33.713284 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:25:33.713407 9921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:33.707323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38243 (local address 127.2.39.193:41249)
0926 06:25:33.707540 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:33.707551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:33.707613 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:33.707752 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:33.707752 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:33.707967 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:33.708033 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:33.708035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.708778 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.708779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:33.709304 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:33.709306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.709766 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.709767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:33.709831 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:33.710265 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:33.710269 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:33.710583 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:33.712172 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:33.712174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:33.712174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:33.712447 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:33.712667 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:33.712668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:33.712669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:33.712749 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:33.713028 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:33.713029 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:33.713076 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:25:33.713182 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 06:25:33.713286 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":75,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab.
Entry for principal oryx/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab.
I20260926 06:25:33.730680 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38243
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab
--principal=oryx/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:33.811736 9923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:33.811982 9923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:33.812031 9923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:33.813231 9923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:33.813300 9923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:33.813337 9923 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:25:33.813393 9923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:33.815062 9923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab
--principal=oryx/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.39.254:38243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:33.815344 9923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:33.815586 9923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:33.820943 9928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:33.820951 9929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:33.821031 9931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:33.821161 9923 server_base.cc:1034] running on GCE node
I20260926 06:25:33.821373 9923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:33.821686 9923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:33.822872 9923 hybrid_clock.cc:648] HybridClock initialized: now 1790403933822843 us; error 37 us; skew 500 ppm
Sep 26 06:25:33 dist-test-slave-fkbn krb5kdc[9666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403933, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:33.825536 9923 init.cc:377] Logged in from keytab as oryx/127.2.39.194@KRBTEST.COM (short username oryx)
I20260926 06:25:33.826040 9923 webserver.cc:460] Webserver started at http://127.2.39.194:41551/ using document root <none> and password file <none>
I20260926 06:25:33.826273 9923 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:33.826347 9923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:33.826478 9923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:33.827749 9923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "ebfaa2736b354548bc3fd47e4833067e"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "d91f854ae033ab9436e627b57e08cc25"
server_key_iv: "dbe36acda42f3404877fd3adf8caed9e"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.828156 9923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "ebfaa2736b354548bc3fd47e4833067e"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "d91f854ae033ab9436e627b57e08cc25"
server_key_iv: "dbe36acda42f3404877fd3adf8caed9e"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.829836 9923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:33.830644 9938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:33.830842 9923 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:33.830971 9923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "ebfaa2736b354548bc3fd47e4833067e"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "d91f854ae033ab9436e627b57e08cc25"
server_key_iv: "dbe36acda42f3404877fd3adf8caed9e"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.831040 9923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:33.854324 9923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:33.856709 9923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:33.856868 9923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:33.857396 9923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:33.857458 9923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:33.857510 9923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:33.857540 9923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:25:33 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403933, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:33.864863 9923 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:33497
I20260926 06:25:33.864993 10053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:33497 every 8 connection(s)
I20260926 06:25:33.865262 9923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:33.865718 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 9923
I20260926 06:25:33.865808 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:33.865967 2207 external_mini_cluster.cc:1428] Setting key f335af60ca1981be1ccc0d9f5422e60f
I20260926 06:25:33.867569 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:33.858018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:39043 (local address 127.2.39.254:38243)
0926 06:25:33.858096 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:33.858099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:33.858343 (+ 244us) server_negotiation.cc:407] Connection header received
0926 06:25:33.859148 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:33.859149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:33.859162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:33.859263 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:33.860200 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.860708 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.861804 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.862020 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.862122 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:33.864382 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:33.864384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:33.864384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:33.864395 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:33.866091 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:33.866494 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:33.866496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:33.866496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:33.866559 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:33.866857 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:33.866858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:33.866858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:33.866966 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:33.867021 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:33.867442 (+ 421us) server_negotiation.cc:299] Negotiation successful
0926 06:25:33.867505 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:25:33.867795 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:33.857994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38243 (local address 127.2.39.194:39043)
0926 06:25:33.858181 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:33.858234 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:33.859007 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:33.859359 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:33.859365 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:33.859785 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:33.860092 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:33.860099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.860810 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.860812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:33.861715 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:33.861718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.862145 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.862146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:33.862260 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:33.862740 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:33.862749 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:33.864282 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:33.866169 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:33.866175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:33.866177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:33.866395 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:33.866620 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:33.866621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:33.866622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:33.866775 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:33.867062 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:33.867068 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:33.867236 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 06:25:33.867350 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 06:25:33.867523 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
I20260926 06:25:33.868316 10054 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38243
I20260926 06:25:33.868397 10054 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:33.868582 10054 heartbeater.cc:511] Master 127.2.39.254:38243 requested a full tablet report, sending...
I20260926 06:25:33.868968 9711 ts_manager.cc:194] Registered new tserver with Master: ebfaa2736b354548bc3fd47e4833067e (127.2.39.194:33497)
I20260926 06:25:33.869443 9711 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:39043
WARNING: no policy specified for oryx/127.2.39.195@KRBTEST.COM; defaulting to no policy
I20260926 06:25:33.876153 10061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:33.869711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38243 (local address 127.2.39.194:33741)
0926 06:25:33.870289 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:33.870300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:33.870364 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:33.870474 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:33.870475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:33.870694 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:33.870782 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:33.870783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.871426 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.871428 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:33.872373 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:33.872376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.872705 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.872706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:33.872776 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:33.873180 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:33.873185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:33.873588 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:33.875253 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:33.875253 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:33.875254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:33.875409 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:33.875591 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:33.875592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:33.875592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:33.875643 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:33.875925 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:33.875929 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:33.875965 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 06:25:33.876019 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 06:25:33.876092 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":500,"threads_started":1}
I20260926 06:25:33.876219 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:33.869770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:33741 (local address 127.2.39.254:38243)
0926 06:25:33.869827 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:33.869829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:33.870400 (+ 571us) server_negotiation.cc:407] Connection header received
0926 06:25:33.870419 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:33.870420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:33.870428 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:33.870465 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:33.870885 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.871325 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.872456 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:33.872634 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:33.872682 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:33.873671 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:33.873673 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:33.873673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:33.873681 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:33.875173 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:33.875483 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:33.875484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:33.875485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:33.875532 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:33.875681 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:33.875682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:33.875682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:33.875799 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:33.875870 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:33.876067 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 06:25:33.876134 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab.
Entry for principal oryx/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab.
I20260926 06:25:33.892717 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38243
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab
--principal=oryx/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:33.976946 10063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:33.977140 10063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:33.977164 10063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:33.978338 10063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:33.978379 10063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:33.978398 10063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 06:25:33.978416 10063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:33.979892 10063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab
--principal=oryx/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.39.254:38243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:33.980141 10063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:33.980366 10063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:33.985276 10069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:33.985270 10068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:33.985352 10063 server_base.cc:1034] running on GCE node
W20260926 06:25:33.985270 10071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:33.985693 10063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:33.985893 10063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:33.987046 10063 hybrid_clock.cc:648] HybridClock initialized: now 1790403933987029 us; error 26 us; skew 500 ppm
Sep 26 06:25:33 dist-test-slave-fkbn krb5kdc[9666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403933, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:33.989497 10063 init.cc:377] Logged in from keytab as oryx/127.2.39.195@KRBTEST.COM (short username oryx)
I20260926 06:25:33.989995 10063 webserver.cc:460] Webserver started at http://127.2.39.195:39153/ using document root <none> and password file <none>
I20260926 06:25:33.990226 10063 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:33.990293 10063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:33.990448 10063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:33.991493 10063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "ad136d80b6f245adbff2a31ee39ae19a"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "88e9a280f202436bda8564462eb1d0ef"
server_key_iv: "fd44ec8d958fdfd947b2e60b54c37770"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.991854 10063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "ad136d80b6f245adbff2a31ee39ae19a"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "88e9a280f202436bda8564462eb1d0ef"
server_key_iv: "fd44ec8d958fdfd947b2e60b54c37770"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.993145 10063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:33.993897 10078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:33.994066 10063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:33.994159 10063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "ad136d80b6f245adbff2a31ee39ae19a"
format_stamp: "Formatted at 2026-09-26 06:25:33 on dist-test-slave-fkbn"
server_key: "88e9a280f202436bda8564462eb1d0ef"
server_key_iv: "fd44ec8d958fdfd947b2e60b54c37770"
server_key_version: "encryptionkey@0"
I20260926 06:25:33.994237 10063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:34.027113 10063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:34.029304 10063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:34.029475 10063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:34.030030 10063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:34.030094 10063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:34.030136 10063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:34.030165 10063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403933, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:34.037130 10063 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:46521
I20260926 06:25:34.037220 10193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:46521 every 8 connection(s)
I20260926 06:25:34.037557 10063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:34.037766 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 10063
I20260926 06:25:34.037909 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:34.038123 2207 external_mini_cluster.cc:1428] Setting key a2c388aad8286941f0af4e6c049bfac5
I20260926 06:25:34.040180 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.030507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:46655 (local address 127.2.39.254:38243)
0926 06:25:34.030583 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.030586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.030794 (+ 208us) server_negotiation.cc:407] Connection header received
0926 06:25:34.031288 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.031289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.031304 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.031392 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:34.032242 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.033078 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.034263 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.034604 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.034688 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.037236 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:34.037238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:34.037238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:34.037246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:34.038814 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:34.039227 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:34.039228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:34.039228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:34.039294 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:34.039548 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:34.039548 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:34.039549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:34.039662 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:34.039730 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.040015 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.040091 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:25:34.040357 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.030497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38243 (local address 127.2.39.195:46655)
0926 06:25:34.030661 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.030699 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.031192 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.031465 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.031469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.031843 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:34.032133 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.032141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.033188 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.033190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.034192 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.034195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.034708 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.034708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.034825 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.035422 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:34.035430 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:34.037128 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:34.038917 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:34.038923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:34.038926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:34.039139 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:34.039365 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:34.039366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:34.039366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:34.039474 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:34.039732 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:34.039738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:34.039871 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.039981 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.040132 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260926 06:25:34.040830 10194 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38243
I20260926 06:25:34.040906 10194 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:34.041096 10194 heartbeater.cc:511] Master 127.2.39.254:38243 requested a full tablet report, sending...
I20260926 06:25:34.041430 9710 ts_manager.cc:194] Registered new tserver with Master: ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521)
I20260926 06:25:34.041826 9710 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:46655
I20260926 06:25:34.046497 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.041876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:53329 (local address 127.2.39.254:38243)
0926 06:25:34.041922 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.041924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.042053 (+ 129us) server_negotiation.cc:407] Connection header received
0926 06:25:34.042101 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.042101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.042111 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.042174 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:34.042507 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.042942 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.043571 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.043761 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.043827 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.044625 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:34.044626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:34.044626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:34.044631 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:34.045737 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:34.046023 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:34.046024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:34.046024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:34.046069 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:34.046181 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:34.046182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:34.046182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:34.046259 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:34.046303 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.046393 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.046442 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 06:25:34.046511 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.041820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38243 (local address 127.2.39.195:53329)
0926 06:25:34.041985 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.041996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.042047 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.042192 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.042193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.042383 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:34.042434 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.042436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.043028 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.043029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.043503 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.043504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.043826 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.043827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.043875 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.044289 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:34.044294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:34.044558 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:34.045806 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:34.045807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:34.045807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:34.045963 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:34.046108 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:34.046108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:34.046109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:34.046149 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:34.046303 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:34.046303 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:34.046334 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.046379 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.046445 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260926 06:25:34.049808 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[9666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:25:34.070101 9779 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:34.064421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44610 (local address 127.2.39.254:38243)
0926 06:25:34.064484 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.064486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.064553 (+ 67us) server_negotiation.cc:407] Connection header received
0926 06:25:34.064652 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.064653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.064664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.064745 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:34.065323 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.065890 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.066641 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.066828 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.066911 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.070018 (+ 3107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44610: BlockingRecv error: recv got EOF from 127.0.0.1:44610 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403934, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:34.077430 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.070844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.2.39.254:38243)
0926 06:25:34.070911 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.070918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.070935 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:25:34.070995 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.070996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.071003 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.071053 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:34.071505 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.071955 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.072599 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.072770 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.072816 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.074862 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:34.074864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:34.074864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:34.074871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:34.076490 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:34.076870 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:34.076871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:34.076872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:34.076938 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:34.077097 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:34.077098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:34.077098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:34.077187 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:34.077254 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.077313 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.077369 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:25:34.078841 9710 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:34.080351 9710 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:34.087169 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.082826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.2.39.195:46521)
0926 06:25:34.083433 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.083437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.083446 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:25:34.083484 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.083485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.083500 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.083573 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:34.083670 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.084529 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.086188 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.086689 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.086789 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.086864 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.087021 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.087059 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":92,"threads_started":1}
I20260926 06:25:34.087173 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.082678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:46521 (local address 127.0.0.1:36482)
0926 06:25:34.082822 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.082861 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.082928 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.083571 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.083571 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.083573 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:34.083614 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.083616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.084612 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.084613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.086122 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.086123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.086793 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.086794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.086887 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.086901 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.087005 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.087061 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":73,"thread_start_us":46,"threads_started":1}
I20260926 06:25:34.087299 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.082900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.2.39.193:44087)
0926 06:25:34.083094 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.083100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.083139 (+ 39us) server_negotiation.cc:407] Connection header received
0926 06:25:34.083187 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.083190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.083206 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.083297 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:34.083480 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.084626 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.086098 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.086798 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.086857 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.086919 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.087162 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.087199 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260926 06:25:34.087435 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.082678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:44087 (local address 127.0.0.1:34168)
0926 06:25:34.083051 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.083059 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.083099 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.083320 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.083323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.083325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:34.083391 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.083392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.084706 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.084707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.086006 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.086008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.086946 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.086947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.087014 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.087018 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.087063 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.087091 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":70,"thread_start_us":91,"threads_started":1}
I20260926 06:25:34.087525 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.082938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.2.39.194:33497)
0926 06:25:34.083139 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.083144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.083154 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:34.083231 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.083232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.083244 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.083328 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:34.083503 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.084669 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.085958 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.086706 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.086781 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.086861 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.087337 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.087403 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260926 06:25:34.087179 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.082688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:33497 (local address 127.0.0.1:58318)
0926 06:25:34.083097 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.083108 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.083156 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.083392 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.083393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.083395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:34.083429 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.083430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.084752 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.084752 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.085865 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.085867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.086897 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.086898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.086960 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.086964 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.087032 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.087078 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":45,"thread_start_us":83,"threads_started":1}
I20260926 06:25:34.088085 9848 tablet_service.cc:1467] Processing CreateTablet for tablet 30ecbd0919554bcca61c4d9310960177 (DEFAULT_TABLE table=test-table [id=a5d9714f0cdb4ba1921d9deeb0517ec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.088086 10128 tablet_service.cc:1467] Processing CreateTablet for tablet 30ecbd0919554bcca61c4d9310960177 (DEFAULT_TABLE table=test-table [id=a5d9714f0cdb4ba1921d9deeb0517ec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.088459 9848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ecbd0919554bcca61c4d9310960177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.088459 10128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ecbd0919554bcca61c4d9310960177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.089964 9988 tablet_service.cc:1467] Processing CreateTablet for tablet 30ecbd0919554bcca61c4d9310960177 (DEFAULT_TABLE table=test-table [id=a5d9714f0cdb4ba1921d9deeb0517ec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.090291 9988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ecbd0919554bcca61c4d9310960177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.092238 10222 tablet_bootstrap.cc:492] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: Bootstrap starting.
I20260926 06:25:34.092478 10223 tablet_bootstrap.cc:492] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: Bootstrap starting.
I20260926 06:25:34.093150 10223 tablet_bootstrap.cc:654] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.093492 10223 log.cc:824] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:34.093868 10224 tablet_bootstrap.cc:492] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: Bootstrap starting.
I20260926 06:25:34.094359 10222 tablet_bootstrap.cc:654] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.094431 10224 tablet_bootstrap.cc:654] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.094647 10222 log.cc:824] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:34.094729 10224 log.cc:824] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:34.095402 10222 tablet_bootstrap.cc:492] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: No bootstrap required, opened a new log
I20260926 06:25:34.095458 10222 ts_tablet_manager.cc:1397] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 06:25:34.095471 10223 tablet_bootstrap.cc:492] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: No bootstrap required, opened a new log
I20260926 06:25:34.095539 10223 ts_tablet_manager.cc:1397] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 06:25:34.096954 10222 raft_consensus.cc:348] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } }
I20260926 06:25:34.097086 10222 raft_consensus.cc:374] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.097108 10222 raft_consensus.cc:729] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad136d80b6f245adbff2a31ee39ae19a, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.097060 10223 raft_consensus.cc:348] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } }
I20260926 06:25:34.097236 10223 raft_consensus.cc:374] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.097230 10222 consensus_queue.cc:260] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } }
I20260926 06:25:34.097286 10223 raft_consensus.cc:729] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048edfaedf2e45869146fcf867329c48, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.097394 10223 consensus_queue.cc:260] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } }
I20260926 06:25:34.097517 10222 ts_tablet_manager.cc:1428] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:25:34.097666 10223 ts_tablet_manager.cc:1428] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:25:34.097687 9914 heartbeater.cc:503] Master 127.2.39.254:38243 was elected leader, sending a full tablet report...
I20260926 06:25:34.097693 10194 heartbeater.cc:503] Master 127.2.39.254:38243 was elected leader, sending a full tablet report...
I20260926 06:25:34.098125 10224 tablet_bootstrap.cc:492] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: No bootstrap required, opened a new log
I20260926 06:25:34.098186 10224 ts_tablet_manager.cc:1397] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 06:25:34.099831 10224 raft_consensus.cc:348] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } }
I20260926 06:25:34.099978 10224 raft_consensus.cc:374] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.100032 10224 raft_consensus.cc:729] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfaa2736b354548bc3fd47e4833067e, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.100138 10224 consensus_queue.cc:260] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } }
I20260926 06:25:34.100386 10224 ts_tablet_manager.cc:1428] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:25:34.100520 10054 heartbeater.cc:503] Master 127.2.39.254:38243 was elected leader, sending a full tablet report...
W20260926 06:25:34.115928 10055 tablet.cc:2369] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:25:34.199713 10230 raft_consensus.cc:482] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:34.199834 10230 raft_consensus.cc:504] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } }
I20260926 06:25:34.200170 10230 leader_election.cc:288] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087), ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521)
W20260926 06:25:34.202034 9915 tablet.cc:2369] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:25:34.203190 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.200412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:46139 (local address 127.2.39.193:44087)
0926 06:25:34.200487 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.200489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.200506 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:25:34.200549 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.200550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.200563 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.200624 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:34.200796 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.201290 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.202444 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.202798 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.202900 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.202910 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.203083 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.203119 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:25:34.203229 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.200345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:44087 (local address 127.2.39.194:46139)
0926 06:25:34.200396 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.200409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.200453 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.200629 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.200630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.200631 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:34.200703 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.200705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.201374 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.201375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.202338 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.202340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.202913 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.202915 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.203007 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.203010 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.203077 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.203118 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 06:25:34.203222 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.200501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:40819 (local address 127.2.39.195:46521)
0926 06:25:34.200561 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.200563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.200763 (+ 200us) server_negotiation.cc:407] Connection header received
0926 06:25:34.200843 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.200845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.200855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.200911 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:34.201068 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.201522 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.202494 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.202868 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.202953 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.202961 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.203118 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.203152 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:25:34.203255 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.200347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:46521 (local address 127.2.39.194:40819)
0926 06:25:34.200687 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.200700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.200766 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.200915 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.200915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.200916 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:34.200996 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.200997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.201628 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.201628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.202406 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.202408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.202954 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.202954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.203034 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.203037 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.203101 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.203141 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":159,"thread_start_us":132,"threads_started":1}
I20260926 06:25:34.203565 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ecbd0919554bcca61c4d9310960177" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048edfaedf2e45869146fcf867329c48" is_pre_election: true
I20260926 06:25:34.203711 9868 raft_consensus.cc:2454] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 0.
I20260926 06:25:34.203799 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ecbd0919554bcca61c4d9310960177" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad136d80b6f245adbff2a31ee39ae19a" is_pre_election: true
I20260926 06:25:34.203946 10148 raft_consensus.cc:2454] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 0.
I20260926 06:25:34.203944 9939 leader_election.cc:302] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048edfaedf2e45869146fcf867329c48, ebfaa2736b354548bc3fd47e4833067e; no voters:
I20260926 06:25:34.204069 10230 raft_consensus.cc:2781] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:34.204146 10230 raft_consensus.cc:482] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:34.204171 10230 raft_consensus.cc:3037] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.205108 10230 raft_consensus.cc:504] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } }
I20260926 06:25:34.205248 10230 leader_election.cc:288] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 election: Requested vote from peers 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087), ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521)
I20260926 06:25:34.205379 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ecbd0919554bcca61c4d9310960177" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048edfaedf2e45869146fcf867329c48"
I20260926 06:25:34.205452 9868 raft_consensus.cc:3037] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.206300 9868 raft_consensus.cc:2454] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 1.
I20260926 06:25:34.206410 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30ecbd0919554bcca61c4d9310960177" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad136d80b6f245adbff2a31ee39ae19a"
I20260926 06:25:34.206452 9939 leader_election.cc:302] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048edfaedf2e45869146fcf867329c48, ebfaa2736b354548bc3fd47e4833067e; no voters:
I20260926 06:25:34.206528 10230 raft_consensus.cc:2781] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:34.206494 10148 raft_consensus.cc:3037] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.206681 10230 raft_consensus.cc:686] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 1 LEADER]: Becoming Leader. State: Replica: ebfaa2736b354548bc3fd47e4833067e, State: Running, Role: LEADER
I20260926 06:25:34.206795 10230 consensus_queue.cc:237] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } }
I20260926 06:25:34.207432 10148 raft_consensus.cc:2454] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 1.
I20260926 06:25:34.207561 9711 catalog_manager.cc:5495] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e reported cstate change: term changed from 0 to 1, leader changed from <none> to ebfaa2736b354548bc3fd47e4833067e (127.2.39.194). New cstate: current_term: 1 leader_uuid: "ebfaa2736b354548bc3fd47e4833067e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } health_report { overall_health: HEALTHY } } }
I20260926 06:25:34.231957 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.229439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58326 (local address 127.2.39.194:33497)
0926 06:25:34.229507 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.229510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.229517 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:34.229559 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.229560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.229625 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.229747 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:34.229930 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.230417 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.231176 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.231381 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.231459 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.231634 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.231739 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.231833 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.231852 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.231875 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:25:34.233952 9868 raft_consensus.cc:1261] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Refusing update from remote peer ebfaa2736b354548bc3fd47e4833067e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:34.234000 10148 raft_consensus.cc:1261] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Refusing update from remote peer ebfaa2736b354548bc3fd47e4833067e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:34.234355 10232 consensus_queue.cc:1035] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [LEADER]: Connected to new peer: Peer: permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:34.234495 10230 consensus_queue.cc:1035] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:34.237927 10237 mvcc.cc:205] Tried to move back new op lower bound from 7333494514619424768 to 7333494514511581184. Current Snapshot: MvccSnapshot[applied={T|T < 7333494514619424768}]
I20260926 06:25:34.238966 10236 mvcc.cc:205] Tried to move back new op lower bound from 7333494514619424768 to 7333494514511581184. Current Snapshot: MvccSnapshot[applied={T|T < 7333494514619424768 or (T in {7333494514619424768})}]
I20260926 06:25:34.238950 9711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:34.239045 9711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:34.239781 9711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961: Sending DeleteTablet for 3 replicas of tablet 30ecbd0919554bcca61c4d9310960177
I20260926 06:25:34.240085 9988 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ecbd0919554bcca61c4d9310960177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:58318
I20260926 06:25:34.240087 9848 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ecbd0919554bcca61c4d9310960177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:34168
I20260926 06:25:34.240263 10244 tablet_replica.cc:330] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: stopping tablet replica
I20260926 06:25:34.240358 10244 raft_consensus.cc:2229] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:34.240469 10244 raft_consensus.cc:2258] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.240444 10128 tablet_service.cc:1514] Processing DeleteTablet for tablet 30ecbd0919554bcca61c4d9310960177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:36482
I20260926 06:25:34.240434 9711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:34.240578 9711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:34.240828 10245 tablet_replica.cc:330] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: stopping tablet replica
I20260926 06:25:34.240909 10245 raft_consensus.cc:2229] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:34.240922 10244 ts_tablet_manager.cc:1905] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:34.240988 10245 raft_consensus.cc:2258] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.241128 10246 tablet_replica.cc:330] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: stopping tablet replica
I20260926 06:25:34.241202 10246 raft_consensus.cc:2229] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:34.241561 10246 raft_consensus.cc:2258] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.242090 10245 ts_tablet_manager.cc:1905] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:34.242272 9848 tablet_service.cc:1467] Processing CreateTablet for tablet c4f7c1592d6e49dfac0a5b1a47bee8ee (DEFAULT_TABLE table=test-table [id=58582dd9edb64a348b6430627791c25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.242378 9848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f7c1592d6e49dfac0a5b1a47bee8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.242841 10244 ts_tablet_manager.cc:1918] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:34.242908 10244 log.cc:1196] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/30ecbd0919554bcca61c4d9310960177
I20260926 06:25:34.243161 10244 ts_tablet_manager.cc:1939] T 30ecbd0919554bcca61c4d9310960177 P 048edfaedf2e45869146fcf867329c48: Deleting consensus metadata
I20260926 06:25:34.243534 9695 catalog_manager.cc:4841] TS 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087): tablet 30ecbd0919554bcca61c4d9310960177 (table test-table [id=a5d9714f0cdb4ba1921d9deeb0517ec5]) successfully deleted
I20260926 06:25:34.243567 10223 tablet_bootstrap.cc:492] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48: Bootstrap starting.
I20260926 06:25:34.243816 9988 tablet_service.cc:1467] Processing CreateTablet for tablet c4f7c1592d6e49dfac0a5b1a47bee8ee (DEFAULT_TABLE table=test-table [id=58582dd9edb64a348b6430627791c25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.243880 10128 tablet_service.cc:1467] Processing CreateTablet for tablet c4f7c1592d6e49dfac0a5b1a47bee8ee (DEFAULT_TABLE table=test-table [id=58582dd9edb64a348b6430627791c25f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.243911 9988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f7c1592d6e49dfac0a5b1a47bee8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.243942 10223 tablet_bootstrap.cc:654] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.243989 10128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f7c1592d6e49dfac0a5b1a47bee8ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.244495 10245 ts_tablet_manager.cc:1918] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:34.244556 10245 log.cc:1196] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/30ecbd0919554bcca61c4d9310960177
I20260926 06:25:34.244556 10223 tablet_bootstrap.cc:492] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48: No bootstrap required, opened a new log
I20260926 06:25:34.244592 10223 ts_tablet_manager.cc:1397] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:34.244822 10245 ts_tablet_manager.cc:1939] T 30ecbd0919554bcca61c4d9310960177 P ebfaa2736b354548bc3fd47e4833067e: Deleting consensus metadata
I20260926 06:25:34.244939 10223 raft_consensus.cc:348] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.245004 10223 raft_consensus.cc:374] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.245024 10223 raft_consensus.cc:729] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048edfaedf2e45869146fcf867329c48, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.245064 10223 consensus_queue.cc:260] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.245158 10222 tablet_bootstrap.cc:492] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: Bootstrap starting.
I20260926 06:25:34.245167 10223 ts_tablet_manager.cc:1428] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:34.245504 10222 tablet_bootstrap.cc:654] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.245786 10246 ts_tablet_manager.cc:1905] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:34.246075 9696 catalog_manager.cc:4841] TS ebfaa2736b354548bc3fd47e4833067e (127.2.39.194:33497): tablet 30ecbd0919554bcca61c4d9310960177 (table test-table [id=a5d9714f0cdb4ba1921d9deeb0517ec5]) successfully deleted
I20260926 06:25:34.246289 10224 tablet_bootstrap.cc:492] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e: Bootstrap starting.
I20260926 06:25:34.246670 10224 tablet_bootstrap.cc:654] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.246752 10222 tablet_bootstrap.cc:492] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: No bootstrap required, opened a new log
I20260926 06:25:34.246801 10222 ts_tablet_manager.cc:1397] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 06:25:34.246955 10222 raft_consensus.cc:348] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.247048 10222 raft_consensus.cc:374] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.247076 10222 raft_consensus.cc:729] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad136d80b6f245adbff2a31ee39ae19a, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.247121 10222 consensus_queue.cc:260] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.247223 10222 ts_tablet_manager.cc:1428] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:34.247239 10246 ts_tablet_manager.cc:1918] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:34.247298 10246 log.cc:1196] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/30ecbd0919554bcca61c4d9310960177
I20260926 06:25:34.247442 10224 tablet_bootstrap.cc:492] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e: No bootstrap required, opened a new log
I20260926 06:25:34.247502 10224 ts_tablet_manager.cc:1397] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:34.247565 10246 ts_tablet_manager.cc:1939] T 30ecbd0919554bcca61c4d9310960177 P ad136d80b6f245adbff2a31ee39ae19a: Deleting consensus metadata
I20260926 06:25:34.247648 10224 raft_consensus.cc:348] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.247735 10224 raft_consensus.cc:374] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.247784 10224 raft_consensus.cc:729] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfaa2736b354548bc3fd47e4833067e, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.247834 10224 consensus_queue.cc:260] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.247934 10224 ts_tablet_manager.cc:1428] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:34.249960 9698 catalog_manager.cc:4841] TS ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521): tablet 30ecbd0919554bcca61c4d9310960177 (table test-table [id=a5d9714f0cdb4ba1921d9deeb0517ec5]) successfully deleted
I20260926 06:25:34.281129 10230 raft_consensus.cc:482] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:34.281236 10230 raft_consensus.cc:504] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.281385 10230 leader_election.cc:288] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087), ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521)
I20260926 06:25:34.281628 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f7c1592d6e49dfac0a5b1a47bee8ee" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad136d80b6f245adbff2a31ee39ae19a" is_pre_election: true
I20260926 06:25:34.281662 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f7c1592d6e49dfac0a5b1a47bee8ee" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048edfaedf2e45869146fcf867329c48" is_pre_election: true
I20260926 06:25:34.281745 9868 raft_consensus.cc:2454] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 0.
I20260926 06:25:34.281745 10148 raft_consensus.cc:2454] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 0.
I20260926 06:25:34.281910 9939 leader_election.cc:302] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048edfaedf2e45869146fcf867329c48, ebfaa2736b354548bc3fd47e4833067e; no voters:
I20260926 06:25:34.282003 10230 raft_consensus.cc:2781] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:34.282040 10230 raft_consensus.cc:482] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:34.282060 10230 raft_consensus.cc:3037] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.282905 10230 raft_consensus.cc:504] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.283035 10230 leader_election.cc:288] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 election: Requested vote from peers 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087), ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521)
I20260926 06:25:34.283216 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f7c1592d6e49dfac0a5b1a47bee8ee" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048edfaedf2e45869146fcf867329c48"
I20260926 06:25:34.283214 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f7c1592d6e49dfac0a5b1a47bee8ee" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad136d80b6f245adbff2a31ee39ae19a"
I20260926 06:25:34.283289 9868 raft_consensus.cc:3037] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.283309 10148 raft_consensus.cc:3037] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.284045 10148 raft_consensus.cc:2454] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 1.
I20260926 06:25:34.284057 9868 raft_consensus.cc:2454] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 1.
I20260926 06:25:34.284226 9939 leader_election.cc:302] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048edfaedf2e45869146fcf867329c48, ebfaa2736b354548bc3fd47e4833067e; no voters:
I20260926 06:25:34.284314 10230 raft_consensus.cc:2781] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:34.284364 10230 raft_consensus.cc:686] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 1 LEADER]: Becoming Leader. State: Replica: ebfaa2736b354548bc3fd47e4833067e, State: Running, Role: LEADER
I20260926 06:25:34.284417 10230 consensus_queue.cc:237] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.285063 9711 catalog_manager.cc:5495] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e reported cstate change: term changed from 0 to 1, leader changed from <none> to ebfaa2736b354548bc3fd47e4833067e (127.2.39.194). New cstate: current_term: 1 leader_uuid: "ebfaa2736b354548bc3fd47e4833067e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } health_report { overall_health: UNKNOWN } } }
W20260926 06:25:34.288141 10195 tablet.cc:2369] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403933, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.254@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:34.318372 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.310303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38243 (local address 127.0.0.1:44632)
0926 06:25:34.310465 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.310480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.310540 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.310698 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.310699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.310993 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:34.311036 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.311038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.311976 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.311977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.312518 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.312519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.313005 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.313006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.313056 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.313689 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:34.313694 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:34.315288 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:34.317060 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:34.317067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:34.317069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:34.317304 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:34.317646 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:34.317647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:34.317647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:34.317716 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:34.318011 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:34.318017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:34.318061 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.318129 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.318227 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260926 06:25:34.318431 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.310374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.2.39.254:38243)
0926 06:25:34.310428 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.310430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.310561 (+ 131us) server_negotiation.cc:407] Connection header received
0926 06:25:34.310592 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.310593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.310604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.310687 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:34.311131 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.311886 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.312613 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.312906 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.312990 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.315398 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:34.315400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:34.315400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:34.315407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:34.316922 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:34.317453 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:34.317454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:34.317455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:34.317515 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:34.317790 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:34.317791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:34.317791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:34.317922 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:34.317997 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.318176 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.318310 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":30}
I20260926 06:25:34.320811 9711 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:44632:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 06:25:34.321076 9711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:34.322466 9988 tablet_service.cc:1467] Processing CreateTablet for tablet 71b7d2d848cf4d5a8d4b4e7a864c301f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ab6039090c7a4543af8e91c46b6c68d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:25:34.322645 9988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b7d2d848cf4d5a8d4b4e7a864c301f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.322603 10128 tablet_service.cc:1467] Processing CreateTablet for tablet 71b7d2d848cf4d5a8d4b4e7a864c301f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ab6039090c7a4543af8e91c46b6c68d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:25:34.322748 10128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b7d2d848cf4d5a8d4b4e7a864c301f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.323832 10222 tablet_bootstrap.cc:492] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: Bootstrap starting.
I20260926 06:25:34.324151 10224 tablet_bootstrap.cc:492] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: Bootstrap starting.
I20260926 06:25:34.324277 10222 tablet_bootstrap.cc:654] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.324259 9848 tablet_service.cc:1467] Processing CreateTablet for tablet 71b7d2d848cf4d5a8d4b4e7a864c301f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ab6039090c7a4543af8e91c46b6c68d5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 06:25:34.324384 9848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b7d2d848cf4d5a8d4b4e7a864c301f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.324555 10224 tablet_bootstrap.cc:654] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.325469 10224 tablet_bootstrap.cc:492] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: No bootstrap required, opened a new log
I20260926 06:25:34.325516 10224 ts_tablet_manager.cc:1397] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:34.325531 10222 tablet_bootstrap.cc:492] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: No bootstrap required, opened a new log
I20260926 06:25:34.325570 10222 ts_tablet_manager.cc:1397] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 06:25:34.325716 10224 raft_consensus.cc:348] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.325743 10222 raft_consensus.cc:348] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.325793 10224 raft_consensus.cc:374] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.325802 10222 raft_consensus.cc:374] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.325819 10224 raft_consensus.cc:729] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfaa2736b354548bc3fd47e4833067e, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.325836 10222 raft_consensus.cc:729] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad136d80b6f245adbff2a31ee39ae19a, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.325862 10224 consensus_queue.cc:260] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.325891 10222 consensus_queue.cc:260] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.325922 10223 tablet_bootstrap.cc:492] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: Bootstrap starting.
I20260926 06:25:34.325959 10224 ts_tablet_manager.cc:1428] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:34.325978 10222 ts_tablet_manager.cc:1428] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:34.326086 10230 tablet_replica.cc:439] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } }
I20260926 06:25:34.326155 10230 tablet_replica.cc:442] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:34.326339 10223 tablet_bootstrap.cc:654] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.326751 10228 tablet_replica.cc:439] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } }
I20260926 06:25:34.326856 10228 tablet_replica.cc:442] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:34.326984 10223 tablet_bootstrap.cc:492] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: No bootstrap required, opened a new log
I20260926 06:25:34.327046 10223 ts_tablet_manager.cc:1397] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:34.327219 10223 raft_consensus.cc:348] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.327279 10223 raft_consensus.cc:374] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.327302 10223 raft_consensus.cc:729] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048edfaedf2e45869146fcf867329c48, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.327342 10223 consensus_queue.cc:260] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.327452 10223 ts_tablet_manager.cc:1428] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:34.327581 10229 tablet_replica.cc:439] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } }
I20260926 06:25:34.327683 10229 tablet_replica.cc:442] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:34.497532 10228 raft_consensus.cc:482] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:34.497668 10228 raft_consensus.cc:504] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.497953 10228 leader_election.cc:288] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087), ebfaa2736b354548bc3fd47e4833067e (127.2.39.194:33497)
I20260926 06:25:34.501717 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.498216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:49337 (local address 127.2.39.193:44087)
0926 06:25:34.498263 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.498265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.498275 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:34.498303 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.498304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.498320 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.498404 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:34.498600 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.499358 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.500868 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.501344 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.501434 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.501443 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.501592 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.501632 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:25:34.501748 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.498138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:44087 (local address 127.2.39.195:49337)
0926 06:25:34.498179 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.498191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.498235 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.498426 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.498427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.498429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:34.498503 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.498504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.499490 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.499491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.500781 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.500784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.501448 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.501450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.501517 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.501520 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.501612 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.501662 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 06:25:34.502005 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.498214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:38339 (local address 127.2.39.194:33497)
0926 06:25:34.498263 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.498265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.498413 (+ 148us) server_negotiation.cc:407] Connection header received
0926 06:25:34.498487 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.498488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.498494 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.498541 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:34.498702 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.499397 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.501071 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.501636 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.501704 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.501711 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.501879 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.501933 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:25:34.502053 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7d2d848cf4d5a8d4b4e7a864c301f" candidate_uuid: "ad136d80b6f245adbff2a31ee39ae19a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048edfaedf2e45869146fcf867329c48" is_pre_election: true
I20260926 06:25:34.502154 9868 raft_consensus.cc:2454] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad136d80b6f245adbff2a31ee39ae19a in term 0.
I20260926 06:25:34.502274 10260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.498138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:33497 (local address 127.2.39.195:38339)
0926 06:25:34.498305 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.498313 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.498426 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.498557 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.498559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.498561 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:34.498627 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.498628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.499490 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.499491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.500977 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.500979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.501709 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.501710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.501779 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.501783 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.502105 (+ 322us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.502158 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":48,"thread_start_us":60,"threads_started":1}
I20260926 06:25:34.502336 10079 leader_election.cc:302] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048edfaedf2e45869146fcf867329c48, ad136d80b6f245adbff2a31ee39ae19a; no voters:
I20260926 06:25:34.502539 10228 raft_consensus.cc:2781] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:34.502574 10228 raft_consensus.cc:482] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:34.502589 10228 raft_consensus.cc:3037] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.502571 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7d2d848cf4d5a8d4b4e7a864c301f" candidate_uuid: "ad136d80b6f245adbff2a31ee39ae19a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebfaa2736b354548bc3fd47e4833067e" is_pre_election: true
I20260926 06:25:34.502737 10008 raft_consensus.cc:2454] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad136d80b6f245adbff2a31ee39ae19a in term 0.
I20260926 06:25:34.503296 10228 raft_consensus.cc:504] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.503435 10228 leader_election.cc:288] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [CANDIDATE]: Term 1 election: Requested vote from peers 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087), ebfaa2736b354548bc3fd47e4833067e (127.2.39.194:33497)
I20260926 06:25:34.503562 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7d2d848cf4d5a8d4b4e7a864c301f" candidate_uuid: "ad136d80b6f245adbff2a31ee39ae19a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048edfaedf2e45869146fcf867329c48"
I20260926 06:25:34.503594 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b7d2d848cf4d5a8d4b4e7a864c301f" candidate_uuid: "ad136d80b6f245adbff2a31ee39ae19a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebfaa2736b354548bc3fd47e4833067e"
I20260926 06:25:34.503640 9868 raft_consensus.cc:3037] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.503661 10008 raft_consensus.cc:3037] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.504328 9868 raft_consensus.cc:2454] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad136d80b6f245adbff2a31ee39ae19a in term 1.
I20260926 06:25:34.504498 10079 leader_election.cc:302] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048edfaedf2e45869146fcf867329c48, ad136d80b6f245adbff2a31ee39ae19a; no voters:
I20260926 06:25:34.504539 10008 raft_consensus.cc:2454] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad136d80b6f245adbff2a31ee39ae19a in term 1.
I20260926 06:25:34.504606 10228 raft_consensus.cc:2781] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:34.504745 10228 raft_consensus.cc:686] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [term 1 LEADER]: Becoming Leader. State: Replica: ad136d80b6f245adbff2a31ee39ae19a, State: Running, Role: LEADER
I20260926 06:25:34.504819 10228 consensus_queue.cc:237] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.505147 10261 tablet_replica.cc:439] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: TxnStatusTablet state changed. Reason: New leader ad136d80b6f245adbff2a31ee39ae19a. Latest consensus state: current_term: 1 leader_uuid: "ad136d80b6f245adbff2a31ee39ae19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } }
I20260926 06:25:34.505232 10261 tablet_replica.cc:442] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:25:34.505548 10263 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 06:25:34.505684 10263 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 06:25:34.505601 9710 catalog_manager.cc:5495] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a reported cstate change: term changed from 0 to 1, leader changed from <none> to ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195). New cstate: current_term: 1 leader_uuid: "ad136d80b6f245adbff2a31ee39ae19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } health_report { overall_health: HEALTHY } } }
I20260926 06:25:34.513715 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.511263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.2.39.195:46521)
0926 06:25:34.511305 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.511307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.511333 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:25:34.511427 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.511428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.511444 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.511507 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:34.511609 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.512102 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.512870 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.513083 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.513165 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.513379 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.513476 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.513552 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.513631 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.513651 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:25:34.513752 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.511211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:46521 (local address 127.0.0.1:36492)
0926 06:25:34.511290 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.511301 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.511353 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.511512 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.511512 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.511513 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:34.511560 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.511562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.512188 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.512189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.512754 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.512755 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.513167 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.513168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.513231 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.513252 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.513549 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.513559 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.513628 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.513674 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 06:25:34.514860 9868 raft_consensus.cc:1261] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Refusing update from remote peer ad136d80b6f245adbff2a31ee39ae19a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:34.514951 10008 raft_consensus.cc:1261] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Refusing update from remote peer ad136d80b6f245adbff2a31ee39ae19a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:34.515097 10261 consensus_queue.cc:1035] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [LEADER]: Connected to new peer: Peer: permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:34.515339 10228 consensus_queue.cc:1035] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:34.516000 10229 tablet_replica.cc:439] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: TxnStatusTablet state changed. Reason: New leader ad136d80b6f245adbff2a31ee39ae19a. Latest consensus state: current_term: 1 leader_uuid: "ad136d80b6f245adbff2a31ee39ae19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } }
I20260926 06:25:34.516152 10229 tablet_replica.cc:442] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:34.516508 10261 tablet_replica.cc:439] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad136d80b6f245adbff2a31ee39ae19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } }
I20260926 06:25:34.516580 10261 tablet_replica.cc:442] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:25:34.517264 10230 tablet_replica.cc:439] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: TxnStatusTablet state changed. Reason: New leader ad136d80b6f245adbff2a31ee39ae19a. Latest consensus state: current_term: 1 leader_uuid: "ad136d80b6f245adbff2a31ee39ae19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } }
I20260926 06:25:34.517350 10230 tablet_replica.cc:442] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:34.517336 10229 tablet_replica.cc:439] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad136d80b6f245adbff2a31ee39ae19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } }
I20260926 06:25:34.517421 10229 tablet_replica.cc:442] T 71b7d2d848cf4d5a8d4b4e7a864c301f P 048edfaedf2e45869146fcf867329c48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:34.518399 10228 tablet_replica.cc:439] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ad136d80b6f245adbff2a31ee39ae19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } }
I20260926 06:25:34.518471 10228 tablet_replica.cc:442] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ad136d80b6f245adbff2a31ee39ae19a: This TxnStatusTablet replica's current role is: LEADER
I20260926 06:25:34.518671 10235 mvcc.cc:205] Tried to move back new op lower bound from 7333494515770773504 to 7333494515732267008. Current Snapshot: MvccSnapshot[applied={T|T < 7333494515770773504}]
I20260926 06:25:34.518764 10230 tablet_replica.cc:439] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ad136d80b6f245adbff2a31ee39ae19a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } } }
I20260926 06:25:34.518836 10230 tablet_replica.cc:442] T 71b7d2d848cf4d5a8d4b4e7a864c301f P ebfaa2736b354548bc3fd47e4833067e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 06:25:34.526299 10270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.523680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:46521 (local address 127.2.39.194:35133)
0926 06:25:34.523790 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.523801 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.523864 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.523991 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.523991 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.523993 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:34.524032 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.524033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.524862 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.524862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.525436 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.525437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.525916 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.525916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.525976 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.525988 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.526173 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.526175 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.526208 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.526246 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":61,"threads_started":1}
I20260926 06:25:34.526296 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.523735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:35133 (local address 127.2.39.195:46521)
0926 06:25:34.523776 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.523778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.523849 (+ 71us) server_negotiation.cc:407] Connection header received
0926 06:25:34.523923 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.523923 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.523934 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.523990 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:34.524068 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.524780 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.525514 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.525842 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.525910 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.526085 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.526117 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.526168 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.526210 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.526229 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:25:34.528519 9868 raft_consensus.cc:1261] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Refusing update from remote peer ebfaa2736b354548bc3fd47e4833067e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:34.528556 10148 raft_consensus.cc:1261] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Refusing update from remote peer ebfaa2736b354548bc3fd47e4833067e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:34.528735 10232 consensus_queue.cc:1035] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:34.528821 10230 consensus_queue.cc:1035] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [LEADER]: Connected to new peer: Peer: permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:34.537523 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.534479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:33497 (local address 127.2.39.195:59329)
0926 06:25:34.534599 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.534609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.534646 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.534802 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.534802 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.534804 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:34.534864 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.534865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.535788 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.535789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.536739 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.536742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.537188 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.537189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.537233 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.537245 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.537412 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.537414 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.537446 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.537476 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 06:25:34.537539 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.534541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:59329 (local address 127.2.39.194:33497)
0926 06:25:34.534711 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.534713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.534719 (+ 6us) server_negotiation.cc:407] Connection header received
0926 06:25:34.534738 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.534739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.534749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.534799 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:34.534931 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.535716 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.536812 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.537116 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.537182 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.537319 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.537354 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.537405 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.537449 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.537469 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260926 06:25:34.550060 9711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:34.550173 9711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:34.550810 9711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961: Sending DeleteTablet for 3 replicas of tablet c4f7c1592d6e49dfac0a5b1a47bee8ee
I20260926 06:25:34.551057 9848 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f7c1592d6e49dfac0a5b1a47bee8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:34168
I20260926 06:25:34.551153 10244 tablet_replica.cc:330] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48: stopping tablet replica
I20260926 06:25:34.551133 10128 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f7c1592d6e49dfac0a5b1a47bee8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:36482
I20260926 06:25:34.551169 9988 tablet_service.cc:1514] Processing DeleteTablet for tablet c4f7c1592d6e49dfac0a5b1a47bee8ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:58318
I20260926 06:25:34.551198 10244 raft_consensus.cc:2229] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:34.551229 10246 tablet_replica.cc:330] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: stopping tablet replica
I20260926 06:25:34.551254 10244 raft_consensus.cc:2258] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.551275 10246 raft_consensus.cc:2229] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:34.551278 10245 tablet_replica.cc:330] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e: stopping tablet replica
I20260926 06:25:34.551308 10246 raft_consensus.cc:2258] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.551342 10245 raft_consensus.cc:2229] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:34.551398 10245 raft_consensus.cc:2258] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.551666 10245 ts_tablet_manager.cc:1905] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:34.551662 10244 ts_tablet_manager.cc:1905] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:34.553264 10245 ts_tablet_manager.cc:1918] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:34.553323 10245 log.cc:1196] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/c4f7c1592d6e49dfac0a5b1a47bee8ee
I20260926 06:25:34.553540 10245 ts_tablet_manager.cc:1939] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ebfaa2736b354548bc3fd47e4833067e: Deleting consensus metadata
I20260926 06:25:34.553547 10244 ts_tablet_manager.cc:1918] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:34.553646 10244 log.cc:1196] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/c4f7c1592d6e49dfac0a5b1a47bee8ee
I20260926 06:25:34.553922 10244 ts_tablet_manager.cc:1939] T c4f7c1592d6e49dfac0a5b1a47bee8ee P 048edfaedf2e45869146fcf867329c48: Deleting consensus metadata
I20260926 06:25:34.554225 9696 catalog_manager.cc:4841] TS ebfaa2736b354548bc3fd47e4833067e (127.2.39.194:33497): tablet c4f7c1592d6e49dfac0a5b1a47bee8ee (table test-table [id=58582dd9edb64a348b6430627791c25f]) successfully deleted
I20260926 06:25:34.554234 9695 catalog_manager.cc:4841] TS 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087): tablet c4f7c1592d6e49dfac0a5b1a47bee8ee (table test-table [id=58582dd9edb64a348b6430627791c25f]) successfully deleted
I20260926 06:25:34.554896 10246 ts_tablet_manager.cc:1905] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:34.556681 10246 ts_tablet_manager.cc:1918] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:34.556762 10246 log.cc:1196] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/c4f7c1592d6e49dfac0a5b1a47bee8ee
I20260926 06:25:34.556970 10246 ts_tablet_manager.cc:1939] T c4f7c1592d6e49dfac0a5b1a47bee8ee P ad136d80b6f245adbff2a31ee39ae19a: Deleting consensus metadata
I20260926 06:25:34.557376 9698 catalog_manager.cc:4841] TS ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521): tablet c4f7c1592d6e49dfac0a5b1a47bee8ee (table test-table [id=58582dd9edb64a348b6430627791c25f]) successfully deleted
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[9666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:25:34.575732 9779 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:34.569696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44644 (local address 127.2.39.254:38243)
0926 06:25:34.569780 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.569782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.569958 (+ 176us) server_negotiation.cc:407] Connection header received
0926 06:25:34.570030 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.570031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.570040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.570084 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:34.570764 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.571272 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.572369 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.572567 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.572631 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.575660 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44644: BlockingRecv error: recv got EOF from 127.0.0.1:44644 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[9666](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM
I20260926 06:25:34.583825 9779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.576560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.2.39.254:38243)
0926 06:25:34.576601 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.576603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.576634 (+ 31us) server_negotiation.cc:407] Connection header received
0926 06:25:34.576687 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.576687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.576696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.576757 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:34.577175 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.577940 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.578648 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.578918 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.578991 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.581057 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:34.581059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:34.581059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:34.581067 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:34.582584 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:34.583036 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:34.583037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:34.583038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:34.583096 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:34.583297 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:34.583298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:34.583298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:34.583410 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:34.583512 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.583650 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.583730 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:25:34.585431 9711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:34.585644 9711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:34.587042 9848 tablet_service.cc:1467] Processing CreateTablet for tablet 981d0262c7554fcbbad195c8462092b3 (DEFAULT_TABLE table=test-table [id=5c5d29b9c75b45a2b508263959958f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.587078 9988 tablet_service.cc:1467] Processing CreateTablet for tablet 981d0262c7554fcbbad195c8462092b3 (DEFAULT_TABLE table=test-table [id=5c5d29b9c75b45a2b508263959958f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.587201 9988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981d0262c7554fcbbad195c8462092b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.587200 9848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981d0262c7554fcbbad195c8462092b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.588274 10128 tablet_service.cc:1467] Processing CreateTablet for tablet 981d0262c7554fcbbad195c8462092b3 (DEFAULT_TABLE table=test-table [id=5c5d29b9c75b45a2b508263959958f12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.588388 10128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 981d0262c7554fcbbad195c8462092b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.588543 10224 tablet_bootstrap.cc:492] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e: Bootstrap starting.
I20260926 06:25:34.588867 10224 tablet_bootstrap.cc:654] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.589455 10224 tablet_bootstrap.cc:492] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e: No bootstrap required, opened a new log
I20260926 06:25:34.589509 10224 ts_tablet_manager.cc:1397] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:34.589661 10223 tablet_bootstrap.cc:492] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48: Bootstrap starting.
I20260926 06:25:34.589692 10224 raft_consensus.cc:348] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.589771 10224 raft_consensus.cc:374] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.589797 10224 raft_consensus.cc:729] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfaa2736b354548bc3fd47e4833067e, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.589834 10224 consensus_queue.cc:260] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.589929 10224 ts_tablet_manager.cc:1428] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:34.590087 10223 tablet_bootstrap.cc:654] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.590452 10222 tablet_bootstrap.cc:492] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a: Bootstrap starting.
I20260926 06:25:34.590843 10222 tablet_bootstrap.cc:654] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.591576 10222 tablet_bootstrap.cc:492] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a: No bootstrap required, opened a new log
I20260926 06:25:34.591624 10222 ts_tablet_manager.cc:1397] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:34.591756 10222 raft_consensus.cc:348] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.591809 10222 raft_consensus.cc:374] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.591821 10222 raft_consensus.cc:729] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad136d80b6f245adbff2a31ee39ae19a, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.591877 10222 consensus_queue.cc:260] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.591980 10222 ts_tablet_manager.cc:1428] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:34.592373 10223 tablet_bootstrap.cc:492] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48: No bootstrap required, opened a new log
I20260926 06:25:34.592425 10223 ts_tablet_manager.cc:1397] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 06:25:34.592584 10223 raft_consensus.cc:348] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.592680 10223 raft_consensus.cc:374] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.592702 10223 raft_consensus.cc:729] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048edfaedf2e45869146fcf867329c48, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.592751 10223 consensus_queue.cc:260] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.592854 10223 ts_tablet_manager.cc:1428] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:34.795754 10273 raft_consensus.cc:482] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:34.795857 10273 raft_consensus.cc:504] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.796073 10273 leader_election.cc:288] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087), ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521)
I20260926 06:25:34.796308 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981d0262c7554fcbbad195c8462092b3" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048edfaedf2e45869146fcf867329c48" is_pre_election: true
I20260926 06:25:34.796447 9868 raft_consensus.cc:2454] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 0.
I20260926 06:25:34.796463 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981d0262c7554fcbbad195c8462092b3" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad136d80b6f245adbff2a31ee39ae19a" is_pre_election: true
I20260926 06:25:34.796612 10148 raft_consensus.cc:2454] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 0.
I20260926 06:25:34.796696 9939 leader_election.cc:302] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048edfaedf2e45869146fcf867329c48, ebfaa2736b354548bc3fd47e4833067e; no voters:
I20260926 06:25:34.796804 10273 raft_consensus.cc:2781] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:34.796841 10273 raft_consensus.cc:482] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:34.796871 10273 raft_consensus.cc:3037] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.797670 10273 raft_consensus.cc:504] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.797808 10273 leader_election.cc:288] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 election: Requested vote from peers 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087), ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521)
I20260926 06:25:34.797960 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981d0262c7554fcbbad195c8462092b3" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad136d80b6f245adbff2a31ee39ae19a"
I20260926 06:25:34.798002 9868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "981d0262c7554fcbbad195c8462092b3" candidate_uuid: "ebfaa2736b354548bc3fd47e4833067e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048edfaedf2e45869146fcf867329c48"
I20260926 06:25:34.798039 10148 raft_consensus.cc:3037] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.798081 9868 raft_consensus.cc:3037] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.798629 10148 raft_consensus.cc:2454] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 1.
I20260926 06:25:34.798650 9868 raft_consensus.cc:2454] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebfaa2736b354548bc3fd47e4833067e in term 1.
I20260926 06:25:34.798828 9942 leader_election.cc:302] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad136d80b6f245adbff2a31ee39ae19a, ebfaa2736b354548bc3fd47e4833067e; no voters:
I20260926 06:25:34.798906 10273 raft_consensus.cc:2781] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:34.798954 10273 raft_consensus.cc:686] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 1 LEADER]: Becoming Leader. State: Replica: ebfaa2736b354548bc3fd47e4833067e, State: Running, Role: LEADER
I20260926 06:25:34.799008 10273 consensus_queue.cc:237] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.799558 9711 catalog_manager.cc:5495] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e reported cstate change: term changed from 0 to 1, leader changed from <none> to ebfaa2736b354548bc3fd47e4833067e (127.2.39.194). New cstate: current_term: 1 leader_uuid: "ebfaa2736b354548bc3fd47e4833067e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:34.817163 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.814765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58338 (local address 127.2.39.194:33497)
0926 06:25:34.814842 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.814844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.814876 (+ 32us) server_negotiation.cc:407] Connection header received
0926 06:25:34.814906 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.814906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.814920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.814978 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:34.815144 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.815678 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.816457 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.816665 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.816739 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.816893 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.816936 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.817006 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.817059 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.817080 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 06:25:34.817966 10148 raft_consensus.cc:1261] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Refusing update from remote peer ebfaa2736b354548bc3fd47e4833067e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:34.817986 9868 raft_consensus.cc:1261] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Refusing update from remote peer ebfaa2736b354548bc3fd47e4833067e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:34.818221 10232 consensus_queue.cc:1035] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:34.818356 10273 consensus_queue.cc:1035] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [LEADER]: Connected to new peer: Peer: permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:34.820418 9711 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:34.820492 9711 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:34.821045 9711 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961: Sending DeleteTablet for 3 replicas of tablet 981d0262c7554fcbbad195c8462092b3
I20260926 06:25:34.821290 9848 tablet_service.cc:1514] Processing DeleteTablet for tablet 981d0262c7554fcbbad195c8462092b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:34168
I20260926 06:25:34.821316 9988 tablet_service.cc:1514] Processing DeleteTablet for tablet 981d0262c7554fcbbad195c8462092b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:58318
I20260926 06:25:34.821401 10245 tablet_replica.cc:330] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e: stopping tablet replica
I20260926 06:25:34.821450 10245 raft_consensus.cc:2229] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:34.821450 10244 tablet_replica.cc:330] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48: stopping tablet replica
I20260926 06:25:34.821506 10245 raft_consensus.cc:2258] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.821506 10244 raft_consensus.cc:2229] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:34.821615 10128 tablet_service.cc:1514] Processing DeleteTablet for tablet 981d0262c7554fcbbad195c8462092b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:36482
I20260926 06:25:34.821555 10244 raft_consensus.cc:2258] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.821709 10246 tablet_replica.cc:330] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a: stopping tablet replica
I20260926 06:25:34.821769 10246 raft_consensus.cc:2229] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:34.821820 10246 raft_consensus.cc:2258] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.821877 9711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 06:25:34.821950 10244 ts_tablet_manager.cc:1905] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 06:25:34.822009 9711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:34.822090 10245 ts_tablet_manager.cc:1905] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:34.823148 10128 tablet_service.cc:1467] Processing CreateTablet for tablet 9d921241718d45b4b7b54a6c3cecbccc (DEFAULT_TABLE table=test-table [id=e791735d15184118a452ae9cff0c84d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.823170 9848 tablet_service.cc:1467] Processing CreateTablet for tablet 9d921241718d45b4b7b54a6c3cecbccc (DEFAULT_TABLE table=test-table [id=e791735d15184118a452ae9cff0c84d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.823252 10128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d921241718d45b4b7b54a6c3cecbccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.823262 9848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d921241718d45b4b7b54a6c3cecbccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.823568 10244 ts_tablet_manager.cc:1918] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:34.823617 10244 log.cc:1196] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/981d0262c7554fcbbad195c8462092b3
I20260926 06:25:34.823830 10244 ts_tablet_manager.cc:1939] T 981d0262c7554fcbbad195c8462092b3 P 048edfaedf2e45869146fcf867329c48: Deleting consensus metadata
I20260926 06:25:34.823845 10245 ts_tablet_manager.cc:1918] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:34.823899 10245 log.cc:1196] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/981d0262c7554fcbbad195c8462092b3
I20260926 06:25:34.824180 10245 ts_tablet_manager.cc:1939] T 981d0262c7554fcbbad195c8462092b3 P ebfaa2736b354548bc3fd47e4833067e: Deleting consensus metadata
I20260926 06:25:34.824524 10246 ts_tablet_manager.cc:1905] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:34.824611 10222 tablet_bootstrap.cc:492] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a: Bootstrap starting.
I20260926 06:25:34.824646 10223 tablet_bootstrap.cc:492] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48: Bootstrap starting.
I20260926 06:25:34.824774 9696 catalog_manager.cc:4841] TS ebfaa2736b354548bc3fd47e4833067e (127.2.39.194:33497): tablet 981d0262c7554fcbbad195c8462092b3 (table test-table [id=5c5d29b9c75b45a2b508263959958f12]) successfully deleted
I20260926 06:25:34.824925 10222 tablet_bootstrap.cc:654] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.825040 10223 tablet_bootstrap.cc:654] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.825206 9988 tablet_service.cc:1467] Processing CreateTablet for tablet 9d921241718d45b4b7b54a6c3cecbccc (DEFAULT_TABLE table=test-table [id=e791735d15184118a452ae9cff0c84d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:34.825292 9988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d921241718d45b4b7b54a6c3cecbccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:34.825392 9695 catalog_manager.cc:4841] TS 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087): tablet 981d0262c7554fcbbad195c8462092b3 (table test-table [id=5c5d29b9c75b45a2b508263959958f12]) successfully deleted
I20260926 06:25:34.825820 10223 tablet_bootstrap.cc:492] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48: No bootstrap required, opened a new log
I20260926 06:25:34.825860 10223 ts_tablet_manager.cc:1397] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:34.825909 10246 ts_tablet_manager.cc:1918] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:34.825959 10246 log.cc:1196] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/981d0262c7554fcbbad195c8462092b3
I20260926 06:25:34.825997 10223 raft_consensus.cc:348] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.826077 10223 raft_consensus.cc:374] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.826099 10223 raft_consensus.cc:729] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048edfaedf2e45869146fcf867329c48, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.826112 10246 ts_tablet_manager.cc:1939] T 981d0262c7554fcbbad195c8462092b3 P ad136d80b6f245adbff2a31ee39ae19a: Deleting consensus metadata
I20260926 06:25:34.826141 10223 consensus_queue.cc:260] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.826224 10223 ts_tablet_manager.cc:1428] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:34.826562 10224 tablet_bootstrap.cc:492] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e: Bootstrap starting.
I20260926 06:25:34.826709 10222 tablet_bootstrap.cc:492] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a: No bootstrap required, opened a new log
I20260926 06:25:34.826748 10222 ts_tablet_manager.cc:1397] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260926 06:25:34.826864 10222 raft_consensus.cc:348] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.826915 10224 tablet_bootstrap.cc:654] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:34.826921 10222 raft_consensus.cc:374] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.826946 10222 raft_consensus.cc:729] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad136d80b6f245adbff2a31ee39ae19a, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.826987 10222 consensus_queue.cc:260] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.827083 9698 catalog_manager.cc:4841] TS ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521): tablet 981d0262c7554fcbbad195c8462092b3 (table test-table [id=5c5d29b9c75b45a2b508263959958f12]) successfully deleted
I20260926 06:25:34.827097 10222 ts_tablet_manager.cc:1428] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:34.827729 10224 tablet_bootstrap.cc:492] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e: No bootstrap required, opened a new log
I20260926 06:25:34.827769 10224 ts_tablet_manager.cc:1397] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:34.827839 10224 raft_consensus.cc:348] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.827884 10224 raft_consensus.cc:374] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:34.827896 10224 raft_consensus.cc:729] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebfaa2736b354548bc3fd47e4833067e, State: Initialized, Role: FOLLOWER
I20260926 06:25:34.827972 10224 consensus_queue.cc:260] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.828068 10224 ts_tablet_manager.cc:1428] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:34.828361 10229 raft_consensus.cc:482] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:34.828428 10229 raft_consensus.cc:504] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.828809 10229 leader_election.cc:288] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebfaa2736b354548bc3fd47e4833067e (127.2.39.194:33497), ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521)
I20260926 06:25:34.831794 10221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.829048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:59169 (local address 127.2.39.194:33497)
0926 06:25:34.829088 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.829090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.829096 (+ 6us) server_negotiation.cc:407] Connection header received
0926 06:25:34.829135 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.829135 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.829144 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.829199 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:34.829445 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.829984 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.831094 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.831442 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.831494 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.831500 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.831674 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.831710 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:25:34.831810 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.828923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:33497 (local address 127.2.39.193:59169)
0926 06:25:34.829012 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.829023 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.829073 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.829235 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.829236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.829238 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:34.829314 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.829316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.830084 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.830086 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.830982 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.830984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.831516 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.831517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.831602 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.831608 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.831664 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.831707 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":53,"thread_start_us":53,"threads_started":1}
I20260926 06:25:34.831897 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.828970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:57865 (local address 127.2.39.195:46521)
0926 06:25:34.829011 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.829013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.829203 (+ 190us) server_negotiation.cc:407] Connection header received
0926 06:25:34.829259 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.829260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.829269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.829319 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:34.829484 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.829994 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.831067 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.831517 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.831596 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.831603 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.831794 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.831833 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:25:34.832060 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d921241718d45b4b7b54a6c3cecbccc" candidate_uuid: "048edfaedf2e45869146fcf867329c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebfaa2736b354548bc3fd47e4833067e" is_pre_election: true
I20260926 06:25:34.832098 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.828931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:46521 (local address 127.2.39.193:57865)
0926 06:25:34.829139 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.829146 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.829210 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.829324 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.829325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.829327 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:34.829411 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.829415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.830095 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.830096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.830980 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.830982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.831620 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.831621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.831702 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.831706 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.831970 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.832011 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":35,"thread_start_us":50,"threads_started":1}
I20260926 06:25:34.832136 10008 raft_consensus.cc:2454] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 048edfaedf2e45869146fcf867329c48 in term 0.
I20260926 06:25:34.832270 9800 leader_election.cc:302] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048edfaedf2e45869146fcf867329c48, ebfaa2736b354548bc3fd47e4833067e; no voters:
I20260926 06:25:34.832391 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d921241718d45b4b7b54a6c3cecbccc" candidate_uuid: "048edfaedf2e45869146fcf867329c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad136d80b6f245adbff2a31ee39ae19a" is_pre_election: true
I20260926 06:25:34.832465 10148 raft_consensus.cc:2454] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 048edfaedf2e45869146fcf867329c48 in term 0.
I20260926 06:25:34.832468 10229 raft_consensus.cc:2781] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:34.832499 10229 raft_consensus.cc:482] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:34.832511 10229 raft_consensus.cc:3037] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.833063 10229 raft_consensus.cc:504] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.833199 10229 leader_election.cc:288] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [CANDIDATE]: Term 1 election: Requested vote from peers ebfaa2736b354548bc3fd47e4833067e (127.2.39.194:33497), ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521)
I20260926 06:25:34.833349 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d921241718d45b4b7b54a6c3cecbccc" candidate_uuid: "048edfaedf2e45869146fcf867329c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad136d80b6f245adbff2a31ee39ae19a"
I20260926 06:25:34.833331 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d921241718d45b4b7b54a6c3cecbccc" candidate_uuid: "048edfaedf2e45869146fcf867329c48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebfaa2736b354548bc3fd47e4833067e"
I20260926 06:25:34.833422 10008 raft_consensus.cc:3037] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.833425 10148 raft_consensus.cc:3037] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:34.833971 10148 raft_consensus.cc:2454] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 048edfaedf2e45869146fcf867329c48 in term 1.
I20260926 06:25:34.833992 10008 raft_consensus.cc:2454] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 048edfaedf2e45869146fcf867329c48 in term 1.
I20260926 06:25:34.834134 9800 leader_election.cc:302] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048edfaedf2e45869146fcf867329c48, ebfaa2736b354548bc3fd47e4833067e; no voters:
I20260926 06:25:34.834226 10229 raft_consensus.cc:2781] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:34.834385 10229 raft_consensus.cc:686] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 1 LEADER]: Becoming Leader. State: Replica: 048edfaedf2e45869146fcf867329c48, State: Running, Role: LEADER
I20260926 06:25:34.834462 10229 consensus_queue.cc:237] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } }
I20260926 06:25:34.835041 9711 catalog_manager.cc:5495] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 048edfaedf2e45869146fcf867329c48 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "048edfaedf2e45869146fcf867329c48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "048edfaedf2e45869146fcf867329c48" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:34.843119 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.840844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34180 (local address 127.2.39.193:44087)
0926 06:25:34.840881 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.840883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.840913 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:25:34.840981 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.840982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.840996 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.841056 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:34.841131 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.841711 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.842364 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.842534 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.842583 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.842770 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.842864 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.842919 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.843016 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.843036 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:25:34.847638 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.845300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:56441 (local address 127.2.39.195:46521)
0926 06:25:34.845361 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.845362 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.845458 (+ 96us) server_negotiation.cc:407] Connection header received
0926 06:25:34.845555 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.845556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.845584 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.845640 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:34.845765 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.846213 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.846943 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.847144 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.847232 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.847398 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.847434 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.847495 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.847548 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.847567 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:25:34.847680 10305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.845219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:46521 (local address 127.2.39.193:56441)
0926 06:25:34.845397 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.845406 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.845483 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.845643 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.845644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.845645 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:34.845704 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.845705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.846291 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.846292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.846850 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.846851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.847235 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.847236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.847298 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.847309 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.847496 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.847498 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.847546 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.847584 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260926 06:25:34.849640 10148 raft_consensus.cc:1261] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Refusing update from remote peer 048edfaedf2e45869146fcf867329c48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:34.849673 10008 raft_consensus.cc:1261] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Refusing update from remote peer 048edfaedf2e45869146fcf867329c48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:34.849929 10301 consensus_queue.cc:1035] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad136d80b6f245adbff2a31ee39ae19a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 46521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:34.850050 10229 consensus_queue.cc:1035] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebfaa2736b354548bc3fd47e4833067e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:34.856431 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.854211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:33443 (local address 127.2.39.193:44087)
0926 06:25:34.854247 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:34.854249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:34.854350 (+ 101us) server_negotiation.cc:407] Connection header received
0926 06:25:34.854371 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:34.854371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:34.854380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:34.854423 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:34.854569 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.855011 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.855726 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.855935 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.856009 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.856136 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.856165 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.856227 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:34.856348 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 06:25:34.856369 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:25:34.856454 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.854172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:44087 (local address 127.2.39.195:33443)
0926 06:25:34.854251 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.854262 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:34.854301 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:34.854457 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:34.854458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:34.854459 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 06:25:34.854492 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.854493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.855103 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.855104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.855645 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:34.855647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:34.856007 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:34.856008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:34.856059 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:34.856062 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 06:25:34.856277 (+ 215us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 06:25:34.856279 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:34.856329 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:25:34.856368 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 06:25:34.862835 9710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:34.862946 9710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:34.863543 9710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f673aa0e2e2450fb7f34bda46c45961: Sending DeleteTablet for 3 replicas of tablet 9d921241718d45b4b7b54a6c3cecbccc
I20260926 06:25:34.863763 10128 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d921241718d45b4b7b54a6c3cecbccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:36482
I20260926 06:25:34.863763 9988 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d921241718d45b4b7b54a6c3cecbccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:58318
I20260926 06:25:34.863807 9848 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d921241718d45b4b7b54a6c3cecbccc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:34 UTC) from {username='oryx', principal='oryx/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:34168
I20260926 06:25:34.863866 10246 tablet_replica.cc:330] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a: stopping tablet replica
I20260926 06:25:34.863897 10245 tablet_replica.cc:330] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e: stopping tablet replica
I20260926 06:25:34.863919 10246 raft_consensus.cc:2229] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:34.863945 10245 raft_consensus.cc:2229] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:34.863951 10246 raft_consensus.cc:2258] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.863982 10245 raft_consensus.cc:2258] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.864054 10244 tablet_replica.cc:330] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48: stopping tablet replica
I20260926 06:25:34.864112 10244 raft_consensus.cc:2229] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:34.864187 10244 raft_consensus.cc:2258] T 9d921241718d45b4b7b54a6c3cecbccc P 048edfaedf2e45869146fcf867329c48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:34.864362 10246 ts_tablet_manager.cc:1905] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:34.864630 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 9783
I20260926 06:25:34.866142 10246 ts_tablet_manager.cc:1918] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 06:25:34.866194 10246 log.cc:1196] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/9d921241718d45b4b7b54a6c3cecbccc
I20260926 06:25:34.866420 10246 ts_tablet_manager.cc:1939] T 9d921241718d45b4b7b54a6c3cecbccc P ad136d80b6f245adbff2a31ee39ae19a: Deleting consensus metadata
I20260926 06:25:34.866816 9698 catalog_manager.cc:4841] TS ad136d80b6f245adbff2a31ee39ae19a (127.2.39.195:46521): tablet 9d921241718d45b4b7b54a6c3cecbccc (table test-table [id=e791735d15184118a452ae9cff0c84d1]) successfully deleted
W20260926 06:25:34.871456 9695 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:44087 (error 108)
I20260926 06:25:34.871493 10245 ts_tablet_manager.cc:1905] T 9d921241718d45b4b7b54a6c3cecbccc P ebfaa2736b354548bc3fd47e4833067e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:34.871747 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 9923
I20260926 06:25:34.873464 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.873299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:44087 (local address 127.0.0.1:34192)
0926 06:25:34.873395 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.873432 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:44087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":51,"threads_started":1}
W20260926 06:25:34.873674 9695 catalog_manager.cc:4584] TS 048edfaedf2e45869146fcf867329c48 (127.2.39.193:44087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d921241718d45b4b7b54a6c3cecbccc: Network error: Client connection negotiation failed: client connection to 127.2.39.193:44087: connect: Connection refused (error 111)
I20260926 06:25:34.877938 10311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:34.877822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:33497 (local address 127.0.0.1:58354)
0926 06:25:34.877875 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:34.877904 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.194:33497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 06:25:34.877961 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 10063
I20260926 06:25:34.883760 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 9679
2026-09-26T06:25:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2630 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[10314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[10314](info): set up 2 sockets
Sep 26 06:25:34 dist-test-slave-fkbn krb5kdc[10314](info): commencing operation
krb5kdc: starting...
W20260926 06:25:35.954196 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:36 dist-test-slave-fkbn krb5kdc[10314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:36.069316 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:39267
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:39267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:36.149853 10327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:36.150059 10327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:36.150080 10327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:36.151233 10327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:36.151275 10327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:36.151290 10327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:36.151329 10327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:36.151356 10327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:36.152894 10327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:39267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:39267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:36.153162 10327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:36.153393 10327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:36.158164 10332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:36.158231 10327 server_base.cc:1034] running on GCE node
W20260926 06:25:36.158164 10335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:36.158176 10333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:36.158790 10327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:36.159088 10327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:36.160233 10327 hybrid_clock.cc:648] HybridClock initialized: now 1790403936160219 us; error 29 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T06:25:36Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1291 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:36 dist-test-slave-fkbn krb5kdc[10340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:36 dist-test-slave-fkbn krb5kdc[10340](info): set up 2 sockets
Sep 26 06:25:36 dist-test-slave-fkbn krb5kdc[10340](info): commencing operation
krb5kdc: starting...
W20260926 06:25:37.254257 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:37 dist-test-slave-fkbn krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:37.372798 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:46787
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:46787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:37.452584 10353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:37.452769 10353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:37.452790 10353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:37.454043 10353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:37.454083 10353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:37.454098 10353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:37.454110 10353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:37.454123 10353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:37.455574 10353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:46787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:46787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:37.455816 10353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:37.456045 10353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:37.460887 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:37.460933 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:37.460968 10358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:37.461051 10353 server_base.cc:1034] running on GCE node
I20260926 06:25:37.461364 10353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:37.461668 10353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:37.462847 10353 hybrid_clock.cc:648] HybridClock initialized: now 1790403937462828 us; error 47 us; skew 500 ppm
Sep 26 06:25:37 dist-test-slave-fkbn krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:37.465199 10353 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:37.465754 10353 webserver.cc:460] Webserver started at http://127.2.39.254:41579/ using document root <none> and password file <none>
I20260926 06:25:37.465981 10353 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:37.466048 10353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:37.466178 10353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:37.467242 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "caf2e7beac8d41969021d0f7798b1c36"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "866fbf676a4279ce780322b023562cbf"
server_key_iv: "b872634c1f832c4d184e56efc83dc2be"
server_key_version: "encryptionkey@0"
I20260926 06:25:37.467602 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "caf2e7beac8d41969021d0f7798b1c36"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "866fbf676a4279ce780322b023562cbf"
server_key_iv: "b872634c1f832c4d184e56efc83dc2be"
server_key_version: "encryptionkey@0"
I20260926 06:25:37.468853 10353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:37.469619 10368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:37.469777 10353 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:37.469872 10353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "caf2e7beac8d41969021d0f7798b1c36"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "866fbf676a4279ce780322b023562cbf"
server_key_iv: "b872634c1f832c4d184e56efc83dc2be"
server_key_version: "encryptionkey@0"
I20260926 06:25:37.469959 10353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:37.489457 10353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:37.491289 10353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:37.491489 10353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:37.495513 10353 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:46787
I20260926 06:25:37.495569 10420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:46787 every 8 connection(s)
I20260926 06:25:37.495919 10353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:37.496718 10421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:37.497162 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 10353
I20260926 06:25:37.497246 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:37.497466 2207 external_mini_cluster.cc:1428] Setting key ac45954d406853e45229089a097c0695
I20260926 06:25:37.499765 10421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: Bootstrap starting.
I20260926 06:25:37.500437 10421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:37.500730 10421 log.cc:824] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:37.501423 10421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: No bootstrap required, opened a new log
I20260926 06:25:37.503710 10421 raft_consensus.cc:348] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } }
I20260926 06:25:37.503883 10421 raft_consensus.cc:374] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:37.503919 10421 raft_consensus.cc:729] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf2e7beac8d41969021d0f7798b1c36, State: Initialized, Role: FOLLOWER
I20260926 06:25:37.504055 10421 consensus_queue.cc:260] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } }
I20260926 06:25:37.504135 10421 raft_consensus.cc:388] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:37.504163 10421 raft_consensus.cc:482] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:37.504216 10421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 06:25:37 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:37.504871 10421 raft_consensus.cc:504] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } }
I20260926 06:25:37.504957 10421 leader_election.cc:302] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caf2e7beac8d41969021d0f7798b1c36; no voters:
I20260926 06:25:37.505172 10421 leader_election.cc:288] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:37.505198 10426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:37.505448 10421 sys_catalog.cc:564] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:37.505656 10426 raft_consensus.cc:686] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 1 LEADER]: Becoming Leader. State: Replica: caf2e7beac8d41969021d0f7798b1c36, State: Running, Role: LEADER
I20260926 06:25:37.505757 10426 consensus_queue.cc:237] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } }
I20260926 06:25:37.506349 10427 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caf2e7beac8d41969021d0f7798b1c36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } } }
I20260926 06:25:37.506428 10427 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:37.506922 10428 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caf2e7beac8d41969021d0f7798b1c36. Latest consensus state: current_term: 1 leader_uuid: "caf2e7beac8d41969021d0f7798b1c36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } } }
I20260926 06:25:37.507002 10428 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [sys.catalog]: This master's current role is: LEADER
W20260926 06:25:37.507341 10443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:25:37.507404 10443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:25:37.507458 10437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:37.507628 10437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:37.508942 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:37.498061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50428 (local address 127.2.39.254:46787)
0926 06:25:37.498248 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:37.498258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:37.498292 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:25:37.498817 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:37.498825 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:37.498883 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:37.499040 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:37.499623 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:37.501003 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:37.501870 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:37.502467 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:37.502599 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:37.505042 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:37.505046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:37.505046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:37.505057 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:37.507284 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:37.507698 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:37.507701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:37.507702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:37.507800 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:37.508026 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:37.508027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:37.508027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:37.508323 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:37.508467 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:37.508590 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 06:25:37.508723 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260926 06:25:37.509404 10437 catalog_manager.cc:1347] Generated new cluster ID: bb1c37fe854c4b9abae76d0be364f8cb
I20260926 06:25:37.509483 10437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:37.526525 10437 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:37.527083 10437 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:37.535280 10437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: Generated new TSK 0
I20260926 06:25:37.535523 10437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:37.570969 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:46787
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:37.668179 10448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:37.668361 10448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:37.668388 10448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:37.669536 10448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:37.669620 10448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:37.669697 10448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:37.671125 10448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:46787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:37.671375 10448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:37.671600 10448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:37.676818 10454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:37.676820 10453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:37.676961 10448 server_base.cc:1034] running on GCE node
W20260926 06:25:37.676887 10456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:37.677229 10448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:37.677488 10448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:37.678658 10448 hybrid_clock.cc:648] HybridClock initialized: now 1790403937678646 us; error 25 us; skew 500 ppm
Sep 26 06:25:37 dist-test-slave-fkbn krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:37.680975 10448 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:37.681376 10448 webserver.cc:460] Webserver started at http://127.2.39.193:35007/ using document root <none> and password file <none>
I20260926 06:25:37.681660 10448 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:37.681728 10448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:37.681877 10448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:37.682864 10448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "0884f394fcab4471ae5574876a538da5"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "a69558a1c8b6273cca499f5689c5086b"
server_key_iv: "3aecd93ab31e04357d94911702ab3917"
server_key_version: "encryptionkey@0"
I20260926 06:25:37.683209 10448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "0884f394fcab4471ae5574876a538da5"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "a69558a1c8b6273cca499f5689c5086b"
server_key_iv: "3aecd93ab31e04357d94911702ab3917"
server_key_version: "encryptionkey@0"
I20260926 06:25:37.684494 10448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:37.685205 10463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:37.685374 10448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:37.685467 10448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "0884f394fcab4471ae5574876a538da5"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "a69558a1c8b6273cca499f5689c5086b"
server_key_iv: "3aecd93ab31e04357d94911702ab3917"
server_key_version: "encryptionkey@0"
I20260926 06:25:37.685560 10448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:37.701337 10448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:37.703104 10448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:37.703263 10448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:37.703471 10448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:37.703766 10448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:37.703819 10448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:37.703850 10448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:37.703877 10448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:37.709770 10448 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:44121
I20260926 06:25:37.709841 10576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:44121 every 8 connection(s)
I20260926 06:25:37.710155 10448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:25:37 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:37.717526 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 10448
I20260926 06:25:37.717695 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:37.717952 2207 external_mini_cluster.cc:1428] Setting key 8cbf728be29c0d16e063b57ca3ef2241
I20260926 06:25:37.718900 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:37.710883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:33299 (local address 127.2.39.254:46787)
0926 06:25:37.710924 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:37.710926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:37.711102 (+ 176us) server_negotiation.cc:407] Connection header received
0926 06:25:37.711570 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:37.711571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:37.711580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:37.711632 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:37.712475 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:37.712969 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:37.713991 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:37.714190 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:37.714246 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:37.716131 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:37.716132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:37.716132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:37.716139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:37.717290 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:37.717889 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:37.717890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:37.717890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:37.717955 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:37.718233 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:37.718234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:37.718234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:37.718345 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:37.718422 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:37.718707 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 06:25:37.718798 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:25:37.719046 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:37.710836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:33299)
0926 06:25:37.710980 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:37.711019 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:37.711467 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:37.711689 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:37.711692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:37.712206 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:37.712380 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:37.712383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:37.713048 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:37.713049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:37.713916 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:37.713919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:37.714265 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:37.714266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:37.714362 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:37.714736 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:37.714743 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:37.716045 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:37.717468 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:37.717473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:37.717474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:37.717818 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:37.718030 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:37.718031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:37.718031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:37.718134 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:37.718413 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:37.718419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:37.718563 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 06:25:37.718674 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 06:25:37.718813 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260926 06:25:37.719607 10577 heartbeater.cc:347] Connected to a master server at 127.2.39.254:46787
I20260926 06:25:37.719708 10577 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:37.719867 10577 heartbeater.cc:511] Master 127.2.39.254:46787 requested a full tablet report, sending...
I20260926 06:25:37.720402 10385 ts_manager.cc:194] Registered new tserver with Master: 0884f394fcab4471ae5574876a538da5 (127.2.39.193:44121)
I20260926 06:25:37.720995 10385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:33299
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:37.743801 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:46787
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:37.823215 10582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:37.823439 10582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:37.823475 10582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:37.824666 10582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:37.824733 10582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:37.824776 10582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:37.826253 10582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:46787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:37.826524 10582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:37.826746 10582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:37.831734 10590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:37.831784 10588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:37.831784 10587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:37.831807 10582 server_base.cc:1034] running on GCE node
I20260926 06:25:37.832106 10582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:37.832300 10582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:37.833537 10582 hybrid_clock.cc:648] HybridClock initialized: now 1790403937833518 us; error 34 us; skew 500 ppm
Sep 26 06:25:37 dist-test-slave-fkbn krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:37.836073 10582 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:37.836583 10582 webserver.cc:460] Webserver started at http://127.2.39.194:37521/ using document root <none> and password file <none>
I20260926 06:25:37.836835 10582 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:37.836895 10582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:37.837078 10582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:37.838308 10582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "777d5a8de07f41918385198ddfaa9df4"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "17903745ef7da29ceb0d0340c198963a"
server_key_iv: "c08fbe9554933ebe117598560882cade"
server_key_version: "encryptionkey@0"
I20260926 06:25:37.838687 10582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "777d5a8de07f41918385198ddfaa9df4"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "17903745ef7da29ceb0d0340c198963a"
server_key_iv: "c08fbe9554933ebe117598560882cade"
server_key_version: "encryptionkey@0"
I20260926 06:25:37.840098 10582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:37.840858 10597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:37.841037 10582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:37.841138 10582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "777d5a8de07f41918385198ddfaa9df4"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "17903745ef7da29ceb0d0340c198963a"
server_key_iv: "c08fbe9554933ebe117598560882cade"
server_key_version: "encryptionkey@0"
I20260926 06:25:37.841241 10582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:37.865427 10582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:37.867686 10582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:37.867880 10582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:37.868180 10582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:37.868522 10582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:37.868583 10582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:37.868630 10582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:37.868660 10582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:37.875385 10582 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:39761
I20260926 06:25:37.875427 10710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:39761 every 8 connection(s)
I20260926 06:25:37.875795 10582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:37.878371 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 10582
I20260926 06:25:37.878491 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:37.878762 2207 external_mini_cluster.cc:1428] Setting key 3dba1d6fc55788b6c127296aebb2bc10
Sep 26 06:25:37 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:37.885823 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:37.876584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:34489 (local address 127.2.39.254:46787)
0926 06:25:37.876628 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:37.876630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:37.876855 (+ 225us) server_negotiation.cc:407] Connection header received
0926 06:25:37.877342 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:37.877343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:37.877351 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:37.877410 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:37.878649 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:37.879230 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:37.880224 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:37.880418 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:37.880502 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:37.882785 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:37.882787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:37.882787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:37.882796 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:37.884436 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:37.884820 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:37.884822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:37.884822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:37.884887 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:37.885122 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:37.885122 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:37.885123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:37.885262 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:37.885339 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:37.885624 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 06:25:37.885712 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:25:37.885942 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:37.876554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:34489)
0926 06:25:37.876717 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:37.876761 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:37.877258 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:37.877465 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:37.877468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:37.878125 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:37.878441 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:37.878448 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:37.879320 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:37.879321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:37.880128 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:37.880130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:37.880512 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:37.880513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:37.880653 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:37.881206 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:37.881214 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:37.882694 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:37.884525 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:37.884530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:37.884532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:37.884745 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:37.884959 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:37.884960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:37.884960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:37.885055 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:37.885333 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:37.885337 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:37.885464 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 06:25:37.885549 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 06:25:37.885732 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260926 06:25:37.886519 10711 heartbeater.cc:347] Connected to a master server at 127.2.39.254:46787
I20260926 06:25:37.886628 10711 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:37.886813 10711 heartbeater.cc:511] Master 127.2.39.254:46787 requested a full tablet report, sending...
I20260926 06:25:37.887219 10385 ts_manager.cc:194] Registered new tserver with Master: 777d5a8de07f41918385198ddfaa9df4 (127.2.39.194:39761)
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
I20260926 06:25:37.887740 10385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:34489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:37.906108 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:46787
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:37.987991 10716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:37.988257 10716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:37.988287 10716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:37.989681 10716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:37.989723 10716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:37.989751 10716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:37.991333 10716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:46787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:37.991572 10716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:37.991765 10716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:37.997149 10722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:37.997260 10716 server_base.cc:1034] running on GCE node
W20260926 06:25:37.997135 10721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:37.997342 10724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:37.997537 10716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:37.997812 10716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:37.998999 10716 hybrid_clock.cc:648] HybridClock initialized: now 1790403937998971 us; error 37 us; skew 500 ppm
Sep 26 06:25:37 dist-test-slave-fkbn krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:38.001431 10716 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:38.001900 10716 webserver.cc:460] Webserver started at http://127.2.39.195:38909/ using document root <none> and password file <none>
I20260926 06:25:38.002157 10716 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:38.002228 10716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:38.002358 10716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:38.003384 10716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "7aad13d27426479a9ae3569efb925bc5"
format_stamp: "Formatted at 2026-09-26 06:25:38 on dist-test-slave-fkbn"
server_key: "65bb714eb733d04b01359432302511df"
server_key_iv: "1345f5465892d57eb45988cd5a5a8110"
server_key_version: "encryptionkey@0"
I20260926 06:25:38.003744 10716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "7aad13d27426479a9ae3569efb925bc5"
format_stamp: "Formatted at 2026-09-26 06:25:38 on dist-test-slave-fkbn"
server_key: "65bb714eb733d04b01359432302511df"
server_key_iv: "1345f5465892d57eb45988cd5a5a8110"
server_key_version: "encryptionkey@0"
I20260926 06:25:38.005018 10716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:38.005828 10731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.006017 10716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:38.006114 10716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "7aad13d27426479a9ae3569efb925bc5"
format_stamp: "Formatted at 2026-09-26 06:25:38 on dist-test-slave-fkbn"
server_key: "65bb714eb733d04b01359432302511df"
server_key_iv: "1345f5465892d57eb45988cd5a5a8110"
server_key_version: "encryptionkey@0"
I20260926 06:25:38.006187 10716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:38.026851 10716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:38.028676 10716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:38.028836 10716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:38.029074 10716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:38.029376 10716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:38.029428 10716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.029469 10716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:38.029492 10716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.035053 10716 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:46749
I20260926 06:25:38.035097 10844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:46749 every 8 connection(s)
I20260926 06:25:38.035512 10716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:38.040967 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 10716
I20260926 06:25:38.041088 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:38.041270 2207 external_mini_cluster.cc:1428] Setting key 4f915b649d19fa612b1fbe181a0f3bf5
I20260926 06:25:38.044430 10424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.036189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:55877 (local address 127.2.39.254:46787)
0926 06:25:38.036266 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.036269 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.036455 (+ 186us) server_negotiation.cc:407] Connection header received
0926 06:25:38.036894 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.036895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.036904 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.036954 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.037895 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.038374 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.039311 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.039484 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.039532 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.041599 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:38.041601 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:38.041601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:38.041608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:38.043008 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:38.043424 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:38.043425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:38.043426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:38.043492 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:38.043754 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:38.043755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:38.043755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:38.043871 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:38.043948 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:38.044254 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 06:25:38.044344 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:25:38.044607 10847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.036158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:55877)
0926 06:25:38.036341 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:38.036376 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:38.036825 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:38.037013 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:38.037016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:38.037561 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:38.037820 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.037826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.038448 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.038449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.039238 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.039241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.039550 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.039550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.039642 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.040046 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:38.040053 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:38.041487 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:38.043100 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:38.043106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:38.043108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:38.043332 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:38.043573 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:38.043573 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:38.043574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:38.043679 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:38.043939 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:38.043945 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:38.044099 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 06:25:38.044218 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:25:38.044368 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260926 06:25:38.045094 10845 heartbeater.cc:347] Connected to a master server at 127.2.39.254:46787
I20260926 06:25:38.045182 10845 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:38.045368 10845 heartbeater.cc:511] Master 127.2.39.254:46787 requested a full tablet report, sending...
I20260926 06:25:38.045768 10385 ts_manager.cc:194] Registered new tserver with Master: 7aad13d27426479a9ae3569efb925bc5 (127.2.39.195:46749)
I20260926 06:25:38.046133 10385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:55877
I20260926 06:25:38.052932 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab.
Entry for principal oryx/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab.
WARNING: no policy specified for oryx/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab.
Entry for principal oryx/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab.
WARNING: no policy specified for oryx/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab.
Entry for principal oryx/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab.
I20260926 06:25:38.124301 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 10448
I20260926 06:25:38.128901 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 10582
I20260926 06:25:38.133479 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 10716
I20260926 06:25:38.137809 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 10353
I20260926 06:25:38.143330 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:46787
--webserver_interface=127.2.39.254
--webserver_port=41579
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:46787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:38.223673 10854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:38.223872 10854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:38.223893 10854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:38.224947 10854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:38.224984 10854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:38.224998 10854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:38.225010 10854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:38.225023 10854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:38.226459 10854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:46787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:46787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=41579
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:38.226668 10854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:38.226832 10854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:38.231738 10860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:38.231731 10862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:38.231731 10859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:38.231935 10854 server_base.cc:1034] running on GCE node
I20260926 06:25:38.232143 10854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:38.232388 10854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:38.233599 10854 hybrid_clock.cc:648] HybridClock initialized: now 1790403938233552 us; error 30 us; skew 500 ppm
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403938, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:38.235989 10854 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:38.236418 10854 webserver.cc:460] Webserver started at http://127.2.39.254:41579/ using document root <none> and password file <none>
I20260926 06:25:38.236666 10854 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:38.236723 10854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:38.238205 10854 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:25:38.238884 10869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.239136 10854 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:38.239236 10854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "caf2e7beac8d41969021d0f7798b1c36"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "866fbf676a4279ce780322b023562cbf"
server_key_iv: "b872634c1f832c4d184e56efc83dc2be"
server_key_version: "encryptionkey@0"
I20260926 06:25:38.239626 10854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:38.247700 10854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:38.250037 10854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:38.250168 10854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:38.253819 10854 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:46787
I20260926 06:25:38.253885 10921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:46787 every 8 connection(s)
I20260926 06:25:38.254236 10854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:38.255645 10922 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:25:38.256258 10922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: Bootstrap starting.
I20260926 06:25:38.258167 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 10854
I20260926 06:25:38.258473 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:44121
--local_ip_for_outbound_sockets=127.2.39.193
--tserver_master_addrs=127.2.39.254:46787
--webserver_port=35007
--webserver_interface=127.2.39.193
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.39.193
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:25:38.264115 10922 log.cc:824] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:38.265795 10922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:25:38.266068 10922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: Bootstrap complete.
I20260926 06:25:38.268210 10922 raft_consensus.cc:348] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } }
I20260926 06:25:38.268446 10922 raft_consensus.cc:729] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf2e7beac8d41969021d0f7798b1c36, State: Initialized, Role: FOLLOWER
I20260926 06:25:38.268632 10922 consensus_queue.cc:260] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } }
I20260926 06:25:38.268710 10922 raft_consensus.cc:388] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:38.268764 10922 raft_consensus.cc:482] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:38.268814 10922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:25:38.269852 10922 raft_consensus.cc:504] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } }
I20260926 06:25:38.269964 10922 leader_election.cc:302] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caf2e7beac8d41969021d0f7798b1c36; no voters:
I20260926 06:25:38.270169 10922 leader_election.cc:288] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:25:38.270239 10926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:25:38.270365 10926 raft_consensus.cc:686] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [term 2 LEADER]: Becoming Leader. State: Replica: caf2e7beac8d41969021d0f7798b1c36, State: Running, Role: LEADER
I20260926 06:25:38.270447 10926 consensus_queue.cc:237] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } }
I20260926 06:25:38.270439 10922 sys_catalog.cc:564] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:38.270704 10928 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caf2e7beac8d41969021d0f7798b1c36. Latest consensus state: current_term: 2 leader_uuid: "caf2e7beac8d41969021d0f7798b1c36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } } }
I20260926 06:25:38.270774 10928 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:38.270747 10927 sys_catalog.cc:455] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "caf2e7beac8d41969021d0f7798b1c36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caf2e7beac8d41969021d0f7798b1c36" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46787 } } }
I20260926 06:25:38.270974 10927 sys_catalog.cc:458] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36 [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:38.271351 10930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:38.271929 10930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:38.272094 10930 catalog_manager.cc:1259] Loaded cluster ID: bb1c37fe854c4b9abae76d0be364f8cb
I20260926 06:25:38.272303 10930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:38.273430 10930 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:38.273617 10930 catalog_manager.cc:5879] T 00000000000000000000000000000000 P caf2e7beac8d41969021d0f7798b1c36: Loaded TSK: 0
I20260926 06:25:38.273839 10930 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:25:38.340096 10924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:38.340327 10924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:38.340361 10924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:38.341532 10924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:38.341625 10924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:38.341682 10924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:38.343196 10924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.193.keytab
--principal=oryx/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:44121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=35007
--tserver_master_addrs=127.2.39.254:46787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:38.343448 10924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:38.343688 10924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:38.349560 10949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:38.349543 10951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:38.349723 10924 server_base.cc:1034] running on GCE node
W20260926 06:25:38.349856 10948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:38.350070 10924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:38.350339 10924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:38.351497 10924 hybrid_clock.cc:648] HybridClock initialized: now 1790403938351482 us; error 31 us; skew 500 ppm
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403938, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:38.354142 10924 init.cc:377] Logged in from keytab as oryx/127.2.39.193@KRBTEST.COM (short username oryx)
I20260926 06:25:38.354645 10924 webserver.cc:460] Webserver started at http://127.2.39.193:35007/ using document root <none> and password file <none>
I20260926 06:25:38.354869 10924 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:38.354951 10924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:38.356416 10924 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:38.357095 10958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.357290 10924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:25:38.357446 10924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "0884f394fcab4471ae5574876a538da5"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "a69558a1c8b6273cca499f5689c5086b"
server_key_iv: "3aecd93ab31e04357d94911702ab3917"
server_key_version: "encryptionkey@0"
I20260926 06:25:38.357846 10924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:38.378486 10924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:38.380702 10924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:38.380873 10924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:38.381115 10924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:38.381487 10924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:38.381543 10924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.381600 10924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:38.381628 10924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.387799 10924 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:44121
I20260926 06:25:38.387853 11071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:44121 every 8 connection(s)
I20260926 06:25:38.388185 10924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:38.393631 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 10924
I20260926 06:25:38.393960 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:39761
--local_ip_for_outbound_sockets=127.2.39.194
--tserver_master_addrs=127.2.39.254:46787
--webserver_port=37521
--webserver_interface=127.2.39.194
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.39.194
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:38.398034 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.388918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:42109)
0926 06:25:38.389128 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:38.389177 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:38.389707 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:38.390165 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:38.390168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:38.390750 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:38.390941 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.390945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.392490 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.392491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.393298 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.393300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.394264 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.394265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.394393 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.394955 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:38.394965 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:38.397796 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":114,"threads_started":1}
W20260926 06:25:38.398329 11072 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:46787 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.39.254:46787: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
W20260926 06:25:38.398706 11075 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:38.388987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:42109 (local address 127.2.39.254:46787)
0926 06:25:38.389292 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.389301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.389345 (+ 44us) server_negotiation.cc:407] Connection header received
0926 06:25:38.389903 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.389910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.389967 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.390128 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.391062 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.392395 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.393465 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.394083 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.394179 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.397939 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:42109: BlockingRecv error: recv got EOF from 127.2.39.193:42109 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":89,"threads_started":1}
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:38.403544 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.398579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:51973)
0926 06:25:38.398638 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:38.398650 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:38.398692 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:38.398920 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:38.398920 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:38.399076 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:38.399126 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.399127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.400127 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.400128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.400611 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.400612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.401118 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.401119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.401165 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.401672 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:38.401683 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:38.403484 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 06:25:38.403703 11077 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:38.398641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:51973 (local address 127.2.39.254:46787)
0926 06:25:38.398789 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.398791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.398800 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:25:38.398823 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.398824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.398836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.398918 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.399212 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.400042 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.400691 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.401040 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.401118 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.403612 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:51973: BlockingRecv error: recv got EOF from 127.2.39.193:51973 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:38.408293 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.403923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:40017)
0926 06:25:38.403970 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:38.403982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:38.404020 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:38.404169 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:38.404169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:38.404269 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:38.404294 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.404295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.405219 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.405220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.405744 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.405746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.406188 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.406188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.406222 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.406617 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:38.406621 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:38.408251 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:25:38.408413 11077 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:38.403991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:40017 (local address 127.2.39.254:46787)
0926 06:25:38.404031 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.404033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.404043 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:25:38.404085 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.404085 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.404096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.404155 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.404371 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.405143 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.405815 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.406117 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.406186 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.408359 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:40017: BlockingRecv error: recv got EOF from 127.2.39.193:40017 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 06:25:38.408456 11072 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:25:38.480108 11076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:38.480309 11076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:38.480331 11076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:38.481490 11076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:38.481529 11076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:38.481555 11076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:38.482986 11076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.194.keytab
--principal=oryx/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:39761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=37521
--tserver_master_addrs=127.2.39.254:46787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:38.483165 11076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:38.483326 11076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:38.488265 11085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:38.488273 11083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:38.488265 11082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:38.488533 11076 server_base.cc:1034] running on GCE node
I20260926 06:25:38.488754 11076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:38.489007 11076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:38.490144 11076 hybrid_clock.cc:648] HybridClock initialized: now 1790403938490122 us; error 36 us; skew 500 ppm
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403938, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:38.492657 11076 init.cc:377] Logged in from keytab as oryx/127.2.39.194@KRBTEST.COM (short username oryx)
I20260926 06:25:38.493098 11076 webserver.cc:460] Webserver started at http://127.2.39.194:37521/ using document root <none> and password file <none>
I20260926 06:25:38.493274 11076 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:38.493323 11076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:38.494649 11076 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:38.495175 11092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.495338 11076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:38.495488 11076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "777d5a8de07f41918385198ddfaa9df4"
format_stamp: "Formatted at 2026-09-26 06:25:37 on dist-test-slave-fkbn"
server_key: "17903745ef7da29ceb0d0340c198963a"
server_key_iv: "c08fbe9554933ebe117598560882cade"
server_key_version: "encryptionkey@0"
I20260926 06:25:38.495816 11076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:38.509959 11076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:38.512246 11076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:38.512424 11076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:38.512665 11076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:38.513003 11076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:38.513054 11076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.513087 11076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:38.513114 11076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.519201 11076 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:39761
I20260926 06:25:38.519245 11205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:39761 every 8 connection(s)
I20260926 06:25:38.519599 11076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 06:25:38.526787 11077 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:38.520326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:55857 (local address 127.2.39.254:46787)
0926 06:25:38.520389 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.520391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.520612 (+ 221us) server_negotiation.cc:407] Connection header received
0926 06:25:38.521079 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.521081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.521093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.521154 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.522091 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.522636 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.523596 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.523782 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.523838 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.526730 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:55857: BlockingRecv error: recv got EOF from 127.2.39.194:55857 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:25:38.526858 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.520297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:55857)
0926 06:25:38.520469 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:38.520511 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:38.520981 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:38.521203 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:38.521206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:38.521786 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:38.521975 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.521980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.522727 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.522728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.523511 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.523512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.523854 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.523855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.523954 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.524422 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:38.524430 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:38.526661 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
W20260926 06:25:38.527117 11206 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:46787 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.39.254:46787: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
I20260926 06:25:38.529646 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 11076
I20260926 06:25:38.529942 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:46749
--local_ip_for_outbound_sockets=127.2.39.195
--tserver_master_addrs=127.2.39.254:46787
--webserver_port=38909
--webserver_interface=127.2.39.195
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.39.195
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:38.531723 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.527351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:37211)
0926 06:25:38.527402 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:38.527414 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:38.527458 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:38.527596 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:38.527596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:38.527729 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:38.527770 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.527772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.528406 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.528407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.528912 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.528913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.529275 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.529276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.529337 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.529804 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:38.529808 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:38.531638 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:25:38.531754 11077 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:38.527451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:37211 (local address 127.2.39.254:46787)
0926 06:25:38.527504 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.527506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.527513 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:38.527533 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.527534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.527545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.527594 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.527847 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.528305 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.529003 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.529207 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.529279 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.531711 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:37211: BlockingRecv error: recv got EOF from 127.2.39.194:37211 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:38.536586 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.532031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:46887)
0926 06:25:38.532144 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:38.532155 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:38.532208 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:38.532354 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:38.532354 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:38.532496 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:38.532529 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.532531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.533124 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.533125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.534017 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.534020 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.534347 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.534349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.534412 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.534773 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:38.534778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:38.536541 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260926 06:25:38.536720 11206 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:25:38.536733 11077 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:38.532100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:46887 (local address 127.2.39.254:46787)
0926 06:25:38.532142 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.532144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.532193 (+ 49us) server_negotiation.cc:407] Connection header received
0926 06:25:38.532267 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.532267 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.532274 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.532316 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.532599 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.533047 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.534108 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.534276 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.534328 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.536651 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:46887: BlockingRecv error: recv got EOF from 127.2.39.194:46887 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 06:25:38.611110 11209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:38.611310 11209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:38.611333 11209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:38.612408 11209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:38.612442 11209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:38.612466 11209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:38.613803 11209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/krb5kdc/oryx_127.2.39.195.keytab
--principal=oryx/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:46749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=38909
--tserver_master_addrs=127.2.39.254:46787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:38.613982 11209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:38.614145 11209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:38.619243 11215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:38.619249 11214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:38.619270 11217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:38.619298 11209 server_base.cc:1034] running on GCE node
I20260926 06:25:38.619674 11209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:38.619920 11209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:38.621062 11209 hybrid_clock.cc:648] HybridClock initialized: now 1790403938621043 us; error 37 us; skew 500 ppm
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403938, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:38.623544 11209 init.cc:377] Logged in from keytab as oryx/127.2.39.195@KRBTEST.COM (short username oryx)
I20260926 06:25:38.623927 11209 webserver.cc:460] Webserver started at http://127.2.39.195:38909/ using document root <none> and password file <none>
I20260926 06:25:38.624112 11209 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:38.624162 11209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:38.625499 11209 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:38.626129 11224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.626278 11209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.626356 11209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "7aad13d27426479a9ae3569efb925bc5"
format_stamp: "Formatted at 2026-09-26 06:25:38 on dist-test-slave-fkbn"
server_key: "65bb714eb733d04b01359432302511df"
server_key_iv: "1345f5465892d57eb45988cd5a5a8110"
server_key_version: "encryptionkey@0"
I20260926 06:25:38.626667 11209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:38.644588 11209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:38.646677 11209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:38.646787 11209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:38.646956 11209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:38.647275 11209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:38.647317 11209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.647341 11209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:38.647353 11209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:38.652801 11209 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:46749
I20260926 06:25:38.652875 11337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:46749 every 8 connection(s)
I20260926 06:25:38.653184 11209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:38.655242 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 11209
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:38.662184 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.653996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:50145)
0926 06:25:38.654197 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:38.654236 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:38.654822 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:38.655173 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:38.655177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:38.656053 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:38.656267 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.656272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.657080 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.657082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.658020 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.658022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.658742 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.658743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.658871 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.659422 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:38.659431 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:38.662042 (+ 2611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":97,"threads_started":1}
W20260926 06:25:38.662218 11077 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:38.654026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:50145 (local address 127.2.39.254:46787)
0926 06:25:38.654102 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.654105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.654420 (+ 315us) server_negotiation.cc:407] Connection header received
0926 06:25:38.654958 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.654960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.654975 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.655054 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.656390 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.656952 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.658142 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.658360 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.658467 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.662138 (+ 3671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:50145: BlockingRecv error: recv got EOF from 127.2.39.195:50145 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 06:25:38.662552 11338 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:46787 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.39.254:46787: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
I20260926 06:25:38.663589 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.655783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.2.39.254:46787)
0926 06:25:38.655832 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.655833 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.655936 (+ 103us) server_negotiation.cc:407] Connection header received
0926 06:25:38.656042 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.656043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.656055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.656102 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.656679 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.657154 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.657960 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.658171 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.658255 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.659559 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:38.659563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:38.659563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:38.659574 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:38.661864 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:38.662343 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:38.662346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:38.662346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:38.662441 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:38.662901 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:38.662902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:38.662903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:38.663121 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:38.663270 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:38.663378 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 06:25:38.663486 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:38.666774 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.662773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:36461)
0926 06:25:38.662820 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:38.662830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:38.662869 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:38.662995 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:38.662995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:38.663117 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:38.663156 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.663157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.663849 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.663850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.664560 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.664562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.664848 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.664849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.664912 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.665225 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:38.665229 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:38.666733 (+ 1504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:25:38.666862 11077 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:38.662823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:36461 (local address 127.2.39.254:46787)
0926 06:25:38.662865 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.662867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.662876 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:25:38.662929 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.662930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.662939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.662981 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.663237 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.663775 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.664620 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.664785 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.664836 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.666807 (+ 1971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:36461: BlockingRecv error: recv got EOF from 127.2.39.195:36461 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:38 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:38.670900 11340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:38.667193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:42413)
0926 06:25:38.667240 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:38.667251 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:38.667298 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:38.667411 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:38.667411 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:38.667510 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:38.667533 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.667534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.668056 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.668057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.668566 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:38.668567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.668847 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.668848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:38.668885 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.669173 (+ 288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:38.669176 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:38.670828 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 06:25:38.670944 11077 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:38.667273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:42413 (local address 127.2.39.254:46787)
0926 06:25:38.667322 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:38.667323 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:38.667331 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:38.667356 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:38.667357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:38.667363 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:38.667406 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:38.667589 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.667990 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.668629 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:38.668790 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:38.668835 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:38.670889 (+ 2054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:42413: BlockingRecv error: recv got EOF from 127.2.39.195:42413 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 06:25:38.671175 11338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 06:25:39 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:39 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:39.413952 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:39.408838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:42613)
0926 06:25:39.408995 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:39.409007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:39.409070 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:39.409249 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:39.409250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:39.409464 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:39.409518 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:39.409520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.410264 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.410266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:39.410843 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:39.410846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.411226 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.411227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:39.411266 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:39.411712 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:39.411721 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:39.413814 (+ 2093us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":54,"threads_started":1}
W20260926 06:25:39.414004 11343 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:39.408985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:42613 (local address 127.2.39.254:46787)
0926 06:25:39.409126 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:39.409128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:39.409135 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:39.409157 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:39.409158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:39.409167 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:39.409226 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:39.409672 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.410167 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.410954 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.411151 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.411220 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:39.413914 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:42613: BlockingRecv error: recv got EOF from 127.2.39.193:42613 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
Sep 26 06:25:39 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:39 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:39.541656 11344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:39.537104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:45979)
0926 06:25:39.537265 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:39.537279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:39.537365 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:39.537524 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:39.537525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:39.537716 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:39.537769 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:39.537771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.538386 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.538387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:39.538977 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:39.538979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.539296 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.539296 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:39.539347 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:39.539812 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:39.539816 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:39.541563 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
W20260926 06:25:39.541764 11343 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:39.537228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:45979 (local address 127.2.39.254:46787)
0926 06:25:39.537298 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:39.537299 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:39.537341 (+ 42us) server_negotiation.cc:407] Connection header received
0926 06:25:39.537439 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:39.537440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:39.537451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:39.537498 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:39.537835 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.538312 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.539051 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.539228 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.539277 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:39.541696 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:45979: BlockingRecv error: recv got EOF from 127.2.39.194:45979 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 06:25:39 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:39 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:39.676296 11345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:39.671584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:51117)
0926 06:25:39.671756 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:39.671768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:39.671832 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:39.672017 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:39.672017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:39.672189 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:39.672235 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:39.672236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.672850 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.672851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:39.673385 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:39.673387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.673906 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.673907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:39.673983 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:39.674534 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:39.674540 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:39.676228 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
W20260926 06:25:39.676476 11343 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:39.671630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:51117 (local address 127.2.39.254:46787)
0926 06:25:39.671692 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:39.671693 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:39.671832 (+ 139us) server_negotiation.cc:407] Connection header received
0926 06:25:39.671919 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:39.671920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:39.671931 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:39.671991 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:39.672298 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.672784 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.673499 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:39.673820 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:39.673895 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:39.676414 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:51117: BlockingRecv error: recv got EOF from 127.2.39.195:51117 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 06:25:40 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:40 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:40.418924 11346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:40.414422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:57107)
0926 06:25:40.414555 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:40.414573 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:40.414620 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:40.414775 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:40.414776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:40.414930 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:40.414977 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:40.414978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.415611 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.415612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:40.416123 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:40.416125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.416551 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.416551 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:40.416589 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:40.417059 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:40.417064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:40.418859 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
W20260926 06:25:40.419073 11347 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:40.414513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:57107 (local address 127.2.39.254:46787)
0926 06:25:40.414656 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:40.414658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:40.414666 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:40.414686 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:40.414687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:40.414697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:40.414760 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:40.415043 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.415548 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.416204 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.416480 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.416537 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:40.418975 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:57107: BlockingRecv error: recv got EOF from 127.2.39.193:57107 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
Sep 26 06:25:40 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:40 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:40.546831 11348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:40.542124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:42831)
0926 06:25:40.542318 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:40.542346 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:40.542427 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:40.542591 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:40.542591 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:40.542766 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:40.542814 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:40.542815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.543493 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.543495 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:40.544049 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:40.544051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.544399 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.544400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:40.544442 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:40.544859 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:40.544863 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:40.546720 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
W20260926 06:25:40.546895 11347 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:40.542220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:42831 (local address 127.2.39.254:46787)
0926 06:25:40.542313 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:40.542315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:40.542430 (+ 115us) server_negotiation.cc:407] Connection header received
0926 06:25:40.542510 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:40.542511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:40.542523 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:40.542571 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:40.542885 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.543405 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.544140 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.544328 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.544384 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:40.546825 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:42831: BlockingRecv error: recv got EOF from 127.2.39.194:42831 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 06:25:40 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:40 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:40.681432 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:40.676695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:42049)
0926 06:25:40.676845 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:40.676862 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:40.676912 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:40.677046 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:40.677047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:40.677207 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:40.677254 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:40.677255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.677888 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.677888 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:40.678452 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:40.678454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.678768 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.678768 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:40.678809 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:40.679239 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:40.679243 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:40.681332 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
W20260926 06:25:40.681457 11347 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:40.676830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:42049 (local address 127.2.39.254:46787)
0926 06:25:40.676897 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:40.676900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:40.676915 (+ 15us) server_negotiation.cc:407] Connection header received
0926 06:25:40.676966 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:40.676966 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:40.676977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:40.677027 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:40.677315 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.677820 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.678520 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:40.678707 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:40.678754 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:40.681412 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:42049: BlockingRecv error: recv got EOF from 127.2.39.195:42049 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 06:25:41 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:41 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:41.429224 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:41.419438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:42907)
0926 06:25:41.419582 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:41.419592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:41.419642 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:41.419812 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:41.419814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:41.419969 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:41.420015 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:41.420016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.420659 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.420660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:41.421185 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:41.421187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.421502 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.421503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:41.421540 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:41.427255 (+ 5715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:41.427265 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:41.429168 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
W20260926 06:25:41.429370 11351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:41.419500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:42907 (local address 127.2.39.254:46787)
0926 06:25:41.419623 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:41.419626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:41.419640 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:25:41.419687 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:41.419688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:41.419698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:41.419757 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:41.420087 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.420589 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.421257 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.421441 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.421489 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:41.429297 (+ 7808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:42907: BlockingRecv error: recv got EOF from 127.2.39.193:42907 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":83,"threads_started":1}
Sep 26 06:25:41 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:41 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:41.551975 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:41.547305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:33825)
0926 06:25:41.547480 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:41.547492 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:41.547598 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:41.547757 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:41.547758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:41.547935 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:41.547981 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:41.547982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.548669 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.548670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:41.549193 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:41.549195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.549633 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.549634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:41.549694 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:41.550166 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:41.550170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:41.551880 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
W20260926 06:25:41.552019 11351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:41.547378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:33825 (local address 127.2.39.254:46787)
0926 06:25:41.547469 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:41.547472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:41.547580 (+ 108us) server_negotiation.cc:407] Connection header received
0926 06:25:41.547689 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:41.547690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:41.547702 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:41.547750 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:41.548083 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.548584 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.549285 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.549506 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.549611 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:41.551958 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:33825: BlockingRecv error: recv got EOF from 127.2.39.194:33825 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 06:25:41 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:41 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:41.686409 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:41.681972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:40203)
0926 06:25:41.682148 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:41.682160 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:41.682220 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:41.682345 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:41.682345 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:41.682501 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:41.682565 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:41.682567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.683207 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.683208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:41.683845 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:41.683848 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.684198 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.684199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:41.684250 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:41.684660 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:41.684664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:41.686345 (+ 1681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":132,"threads_started":1}
W20260926 06:25:41.686520 11351 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:41.682060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:40203 (local address 127.2.39.254:46787)
0926 06:25:41.682131 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:41.682133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:41.682221 (+ 88us) server_negotiation.cc:407] Connection header received
0926 06:25:41.682273 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:41.682274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:41.682285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:41.682327 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:41.682625 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.683116 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.683937 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:41.684119 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:41.684174 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:41.686435 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:40203: BlockingRecv error: recv got EOF from 127.2.39.195:40203 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 06:25:42 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:42 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:42.434516 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:42.429828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:44823)
0926 06:25:42.429972 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:42.429989 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:42.430049 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:42.430170 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:42.430170 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:42.430482 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:42.430530 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:42.430532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.431176 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.431176 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:42.431736 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:42.431738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.432065 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.432065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:42.432108 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:42.432694 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:42.432698 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:42.434458 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
W20260926 06:25:42.434648 11355 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:42.429874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:44823 (local address 127.2.39.254:46787)
0926 06:25:42.429991 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:42.429993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:42.430049 (+ 56us) server_negotiation.cc:407] Connection header received
0926 06:25:42.430089 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:42.430090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:42.430100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:42.430161 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:42.430599 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.431102 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.431808 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.431994 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.432062 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:42.434559 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:44823: BlockingRecv error: recv got EOF from 127.2.39.193:44823 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
Sep 26 06:25:42 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:42 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:42.557205 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:42.552505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:51831)
0926 06:25:42.552650 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:42.552660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:42.552713 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:42.552908 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:42.552910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:42.553067 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:42.553114 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:42.553116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.553798 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.553798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:42.554312 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:42.554313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.554624 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.554624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:42.554662 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:42.555191 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:42.555196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:42.557148 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
W20260926 06:25:42.557274 11355 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:42.552578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:51831 (local address 127.2.39.254:46787)
0926 06:25:42.552674 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:42.552676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:42.552702 (+ 26us) server_negotiation.cc:407] Connection header received
0926 06:25:42.552774 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:42.552775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:42.552786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:42.552845 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:42.553189 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.553684 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.554386 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.554561 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.554609 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:42.557230 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:51831: BlockingRecv error: recv got EOF from 127.2.39.194:51831 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 06:25:42 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:42 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:42.691666 11357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:42.686888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:50775)
0926 06:25:42.687071 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:42.687089 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:42.687166 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:42.687352 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:42.687353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:42.687594 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:42.687652 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:42.687655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.688473 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.688474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:42.689003 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:42.689005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.689325 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.689326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:42.689363 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:42.689832 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:42.689836 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:42.691571 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":94,"threads_started":1}
W20260926 06:25:42.691704 11355 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:42.686990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:50775 (local address 127.2.39.254:46787)
0926 06:25:42.687041 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:42.687043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:42.687154 (+ 111us) server_negotiation.cc:407] Connection header received
0926 06:25:42.687251 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:42.687251 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:42.687263 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:42.687320 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:42.687741 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.688404 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.689082 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:42.689261 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:42.689314 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:42.691645 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:50775: BlockingRecv error: recv got EOF from 127.2.39.195:50775 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 06:25:43 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:43 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:43.440107 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:43.435072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:36611)
0926 06:25:43.435243 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:43.435253 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:43.435313 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:43.435498 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:43.435499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:43.435668 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:43.435741 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:43.435743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.436624 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.436626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:43.437139 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:43.437141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.437518 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.437519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:43.437644 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:43.438135 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:43.438139 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:43.440003 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
W20260926 06:25:43.440160 11359 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:43.435166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:36611 (local address 127.2.39.254:46787)
0926 06:25:43.435284 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:43.435289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:43.435302 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:25:43.435404 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:43.435404 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:43.435417 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:43.435485 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:43.435858 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.436558 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.437214 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.437453 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.437509 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:43.440085 (+ 2576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:36611: BlockingRecv error: recv got EOF from 127.2.39.193:36611 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":53,"threads_started":1}
Sep 26 06:25:43 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:43 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:43.562368 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:43.557714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:56115)
0926 06:25:43.557891 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:43.557902 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:43.557963 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:43.558105 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:43.558106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:43.558266 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:43.558313 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:43.558315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.559020 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.559021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:43.559538 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:43.559540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.559870 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.559871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:43.559908 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:43.560398 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:43.560403 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:43.562311 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
W20260926 06:25:43.562569 11359 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:43.557833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:56115 (local address 127.2.39.254:46787)
0926 06:25:43.557917 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:43.557919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:43.557948 (+ 29us) server_negotiation.cc:407] Connection header received
0926 06:25:43.558023 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:43.558024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:43.558035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:43.558085 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:43.558446 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.558950 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.559612 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.559805 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.559860 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:43.562504 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:56115: BlockingRecv error: recv got EOF from 127.2.39.194:56115 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 06:25:43 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:43 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:43.697090 11361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:43.692292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:41649)
0926 06:25:43.692428 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:43.692447 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:43.692503 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:43.692665 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:43.692666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:43.692818 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:43.692864 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:43.692866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.693549 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.693549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:43.694155 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:43.694157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.694523 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.694534 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:43.694572 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:43.695204 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:43.695210 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:43.697036 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260926 06:25:43.697211 11359 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:43.692362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:41649 (local address 127.2.39.254:46787)
0926 06:25:43.692446 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:43.692448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:43.692504 (+ 56us) server_negotiation.cc:407] Connection header received
0926 06:25:43.692571 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:43.692572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:43.692584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:43.692641 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:43.692972 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.693472 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.694233 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:43.694457 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:43.694514 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:43.697134 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:41649: BlockingRecv error: recv got EOF from 127.2.39.195:41649 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 06:25:44 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:44 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:44.445875 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:44.440661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:50775)
0926 06:25:44.440830 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:44.440841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:44.440904 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:44.441079 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:44.441079 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:44.441231 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:44.441280 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:44.441281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.442052 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.442053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:44.442569 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:44.442570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.443104 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.443105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:44.443166 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:44.443860 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:44.443864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:44.445777 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
W20260926 06:25:44.445928 11363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:44.440770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:50775 (local address 127.2.39.254:46787)
0926 06:25:44.440917 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:44.440919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:44.440944 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:25:44.440968 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:44.440969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:44.440980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:44.441033 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:44.441351 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.441990 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.442667 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.443011 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.443099 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:44.445857 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:50775: BlockingRecv error: recv got EOF from 127.2.39.193:50775 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":51,"threads_started":1}
Sep 26 06:25:44 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:44 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:44.567425 11364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:44.562767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:43401)
0926 06:25:44.562928 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:44.562940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:44.563012 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:44.563138 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:44.563139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:44.563309 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:44.563411 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:44.563413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.564057 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.564059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:44.564642 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:44.564645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.564959 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.564960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:44.564999 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:44.565467 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:44.565472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:44.567340 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
W20260926 06:25:44.567474 11363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:44.562833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:43401 (local address 127.2.39.254:46787)
0926 06:25:44.562895 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:44.562896 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:44.562995 (+ 99us) server_negotiation.cc:407] Connection header received
0926 06:25:44.563059 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:44.563060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:44.563071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:44.563119 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:44.563477 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.563968 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.564722 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.564898 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.564949 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:44.567427 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:43401: BlockingRecv error: recv got EOF from 127.2.39.194:43401 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 06:25:44 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:44 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:44.702359 11365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:44.697606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:49015)
0926 06:25:44.697730 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:44.697740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:44.697820 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:44.697966 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:44.697967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:44.698128 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:44.698172 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:44.698173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.699043 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.699044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:44.699632 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:44.699634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.699967 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.699968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:44.700006 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:44.700513 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:44.700518 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:44.702292 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
W20260926 06:25:44.702581 11363 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:44.697665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:49015 (local address 127.2.39.254:46787)
0926 06:25:44.697726 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:44.697727 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:44.697824 (+ 97us) server_negotiation.cc:407] Connection header received
0926 06:25:44.697881 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:44.697882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:44.697903 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:44.697946 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:44.698325 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.698950 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.699715 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:44.699904 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:44.699959 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:44.702490 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:49015: BlockingRecv error: recv got EOF from 127.2.39.195:49015 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 06:25:45 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:45 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:45.450889 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:45.446404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:38035)
0926 06:25:45.446530 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:45.446541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:45.446596 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:45.446729 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:45.446730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:45.446884 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:45.446930 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:45.446932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.447551 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.447552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:45.448073 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:45.448075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.448477 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.448478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:45.448518 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:45.448937 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:45.448942 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:45.450808 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
W20260926 06:25:45.450959 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:45.446472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:38035 (local address 127.2.39.254:46787)
0926 06:25:45.446615 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:45.446616 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:45.446623 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:45.446643 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:45.446643 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:45.446654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:45.446713 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:45.446993 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.447482 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.448143 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.448412 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.448462 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:45.450898 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:38035: BlockingRecv error: recv got EOF from 127.2.39.193:38035 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
Sep 26 06:25:45 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:45 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:45.572924 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:45.568010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:60739)
0926 06:25:45.568179 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:45.568191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:45.568266 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:45.568408 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:45.568410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:45.568604 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:45.568651 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:45.568652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.569338 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.569339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:45.569892 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:45.569893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.570271 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.570272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:45.570332 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:45.570848 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:45.570854 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:45.572841 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
W20260926 06:25:45.572996 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:45.568092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:60739 (local address 127.2.39.254:46787)
0926 06:25:45.568180 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:45.568182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:45.568259 (+ 77us) server_negotiation.cc:407] Connection header received
0926 06:25:45.568323 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:45.568324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:45.568337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:45.568395 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:45.568735 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.569245 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.569977 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.570183 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.570268 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:45.572923 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:60739: BlockingRecv error: recv got EOF from 127.2.39.194:60739 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 06:25:45 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:45 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:45.707576 11369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:45.702817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:48935)
0926 06:25:45.702948 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:45.702959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:45.703013 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:45.703198 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:45.703198 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:45.703354 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:45.703400 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:45.703401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.704070 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.704071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:45.704592 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:45.704593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.704948 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.704948 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:45.704985 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:45.705431 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:45.705436 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:45.707471 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
W20260926 06:25:45.707635 11367 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:45.702897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:48935 (local address 127.2.39.254:46787)
0926 06:25:45.703074 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:45.703076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:45.703091 (+ 15us) server_negotiation.cc:407] Connection header received
0926 06:25:45.703113 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:45.703114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:45.703126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:45.703179 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:45.703466 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.704004 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.704674 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:45.704880 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:45.704942 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:45.707567 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:48935: BlockingRecv error: recv got EOF from 127.2.39.195:48935 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Sep 26 06:25:46 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:46 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:46.468256 11370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:46.451439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:59927)
0926 06:25:46.451582 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:46.451593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:46.451639 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:46.451847 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:46.451848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:46.452015 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:46.452066 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:46.452068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.452716 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.452717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:46.453243 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:46.453245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.453606 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.453608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:46.453660 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:46.466222 (+ 12562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:46.466228 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:46.468198 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
W20260926 06:25:46.468477 11371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:46.451598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:59927 (local address 127.2.39.254:46787)
0926 06:25:46.451715 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:46.451722 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:46.451729 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:25:46.451750 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:46.451750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:46.451760 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:46.451821 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:46.452139 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.452641 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.453312 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.453491 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.453539 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:46.468320 (+ 14781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:59927: BlockingRecv error: recv got EOF from 127.2.39.193:59927 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
Sep 26 06:25:46 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:46 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:46.578646 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:46.573410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:38369)
0926 06:25:46.573551 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:46.573563 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:46.573673 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:46.573906 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:46.573908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:46.574154 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:46.574204 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:46.574206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.575163 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.575165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:46.575768 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:46.575770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.576221 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.576222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:46.576286 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:46.576747 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:46.576752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:46.578542 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
W20260926 06:25:46.578703 11371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:46.573492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:38369 (local address 127.2.39.254:46787)
0926 06:25:46.573629 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:46.573632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:46.573651 (+ 19us) server_negotiation.cc:407] Connection header received
0926 06:25:46.573751 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:46.573753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:46.573773 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:46.573868 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:46.574296 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.575042 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.575888 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.576105 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.576222 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:46.578636 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:38369: BlockingRecv error: recv got EOF from 127.2.39.194:38369 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 06:25:46 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:46 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:46.712630 11373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:46.708208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:44141)
0926 06:25:46.708362 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:46.708379 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:46.708427 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:46.708580 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:46.708581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:46.708737 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:46.708782 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:46.708784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.709454 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.709455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:46.709991 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:46.709992 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.710317 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.710317 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:46.710355 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:46.710877 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:46.710881 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:46.712575 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
W20260926 06:25:46.712779 11371 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:46.708303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:44141 (local address 127.2.39.254:46787)
0926 06:25:46.708377 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:46.708379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:46.708418 (+ 39us) server_negotiation.cc:407] Connection header received
0926 06:25:46.708488 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:46.708489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:46.708501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:46.708563 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:46.708843 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.709344 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.710075 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:46.710257 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:46.710306 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:46.712695 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:44141: BlockingRecv error: recv got EOF from 127.2.39.195:44141 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 06:25:47 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:47 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:47.476750 11374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:47.468811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:45053)
0926 06:25:47.468960 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:47.468972 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:47.469033 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:47.469184 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:47.469184 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:47.469348 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:47.469396 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:47.469397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.470150 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.470152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:47.470710 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:47.470712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.471054 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.471055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:47.471145 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:47.474772 (+ 3627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:47.474778 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:47.476679 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
W20260926 06:25:47.476831 11375 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:47.468898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:45053 (local address 127.2.39.254:46787)
0926 06:25:47.469059 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:47.469060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:47.469069 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:25:47.469090 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:47.469090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:47.469100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:47.469164 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:47.469463 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.469986 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.470796 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.470987 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.471046 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:47.476778 (+ 5732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:45053: BlockingRecv error: recv got EOF from 127.2.39.193:45053 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
Sep 26 06:25:47 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:47 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:47.584411 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:47.579313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:52655)
0926 06:25:47.579473 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:47.579507 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:47.579574 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:47.579753 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:47.579754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:47.579974 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:47.580024 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:47.580026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.580771 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.580772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:47.581317 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:47.581319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.581837 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.581838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:47.581898 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:47.582434 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:47.582439 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:47.584283 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
W20260926 06:25:47.584452 11375 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:47.579407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:52655 (local address 127.2.39.254:46787)
0926 06:25:47.579474 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:47.579476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:47.579582 (+ 106us) server_negotiation.cc:407] Connection header received
0926 06:25:47.579629 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:47.579630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:47.579646 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:47.579748 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:47.580114 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.580670 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.581476 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.581745 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.581839 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:47.584384 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:52655: BlockingRecv error: recv got EOF from 127.2.39.194:52655 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 06:25:47 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:47 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:47.717952 11377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:47.713070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:57317)
0926 06:25:47.713211 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:47.713229 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:47.713286 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:47.713458 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:47.713460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:47.713657 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:47.713703 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:47.713705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.714338 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.714339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:47.714924 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:47.714926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.715279 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.715280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:47.715382 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:47.715834 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:47.715839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:47.717889 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
W20260926 06:25:47.718020 11375 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:47.713142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:57317 (local address 127.2.39.254:46787)
0926 06:25:47.713189 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:47.713191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:47.713288 (+ 97us) server_negotiation.cc:407] Connection header received
0926 06:25:47.713328 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:47.713329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:47.713340 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:47.713390 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:47.713776 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.714263 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.714998 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:47.715181 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:47.715232 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:47.717973 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:57317: BlockingRecv error: recv got EOF from 127.2.39.195:57317 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 06:25:48 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:48 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:48.482143 11378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:48.477329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:34569)
0926 06:25:48.477504 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:48.477515 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:48.477610 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:48.477766 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:48.477767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:48.477932 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:48.477980 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:48.477981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.478897 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.478899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:48.479435 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:48.479437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.479755 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.479756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:48.479794 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:48.480204 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:48.480208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:48.482054 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":59,"threads_started":1}
W20260926 06:25:48.482189 11379 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:48.477391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:34569 (local address 127.2.39.254:46787)
0926 06:25:48.477521 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:48.477522 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:48.477562 (+ 40us) server_negotiation.cc:407] Connection header received
0926 06:25:48.477668 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:48.477669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:48.477687 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:48.477749 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:48.478058 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.478830 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.479504 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.479693 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.479742 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:48.482139 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:34569: BlockingRecv error: recv got EOF from 127.2.39.193:34569 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
Sep 26 06:25:48 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:48 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:48.589908 11380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:48.584973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:37475)
0926 06:25:48.585173 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:48.585184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:48.585256 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:48.585405 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:48.585406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:48.585647 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:48.585697 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:48.585698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.586639 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.586640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:48.587237 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:48.587239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.587548 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.587549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:48.587597 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:48.588027 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:48.588031 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:48.589827 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":112,"threads_started":1}
W20260926 06:25:48.589959 11379 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:48.585053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:37475 (local address 127.2.39.254:46787)
0926 06:25:48.585136 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:48.585138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:48.585254 (+ 116us) server_negotiation.cc:407] Connection header received
0926 06:25:48.585305 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:48.585306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:48.585317 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:48.585369 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:48.585763 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.586572 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.587308 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.587482 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.587542 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:48.589913 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:37475: BlockingRecv error: recv got EOF from 127.2.39.194:37475 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 06:25:48 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:48 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:48.722797 11381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:48.718542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:54117)
0926 06:25:48.718725 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:48.718743 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:48.718800 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:48.718918 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:48.718919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:48.719081 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:48.719128 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:48.719130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.719745 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.719746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:48.720264 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:48.720265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.720558 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.720558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:48.720595 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:48.721009 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:48.721013 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:48.722742 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
W20260926 06:25:48.722931 11379 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:48.718684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:54117 (local address 127.2.39.254:46787)
0926 06:25:48.718753 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:48.718755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:48.718785 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:25:48.718852 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:48.718853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:48.718863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:48.718906 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:48.719181 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.719675 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.720328 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:48.720497 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:48.720555 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:48.722845 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:54117: BlockingRecv error: recv got EOF from 127.2.39.195:54117 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 06:25:49 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:49 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:49.487447 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:49.482664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:43125)
0926 06:25:49.482831 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:49.482845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:49.482969 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:49.483110 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:49.483111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:49.483286 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:49.483343 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:49.483345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.484001 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.484002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:49.484644 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:49.484647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.484980 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.484981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:49.485026 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:49.485461 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:49.485465 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:49.487352 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":217,"threads_started":1}
W20260926 06:25:49.487490 11383 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:49.482734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:43125 (local address 127.2.39.254:46787)
0926 06:25:49.482865 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:49.482866 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:49.482939 (+ 73us) server_negotiation.cc:407] Connection header received
0926 06:25:49.483016 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:49.483017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:49.483027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:49.483088 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:49.483411 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.483915 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.484731 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.484912 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.484962 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:49.487426 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:43125: BlockingRecv error: recv got EOF from 127.2.39.193:43125 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
Sep 26 06:25:49 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:49 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:49.596325 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:49.590442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:32793)
0926 06:25:49.590665 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:49.590679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:49.590755 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:49.590979 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:49.590980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:49.591177 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:49.591226 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:49.591234 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.591946 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.591948 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:49.592537 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:49.592540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.592936 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.592937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:49.593006 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:49.593677 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:49.593683 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:49.596219 (+ 2536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
W20260926 06:25:49.596378 11383 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:49.590532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:32793 (local address 127.2.39.254:46787)
0926 06:25:49.590602 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:49.590603 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:49.590740 (+ 137us) server_negotiation.cc:407] Connection header received
0926 06:25:49.590888 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:49.590889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:49.590901 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:49.590971 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:49.591322 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.591848 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.592633 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.592840 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.592935 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:49.596311 (+ 3376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:32793: BlockingRecv error: recv got EOF from 127.2.39.194:32793 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 06:25:49 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:49 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:49.727883 11385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:49.723297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:58799)
0926 06:25:49.723452 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:49.723463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:49.723528 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:49.723662 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:49.723662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:49.723847 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:49.723893 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:49.723895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.724569 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.724570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:49.725118 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:49.725120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.725432 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.725433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:49.725473 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:49.725905 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:49.725909 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:49.727818 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
W20260926 06:25:49.728008 11383 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:49.723352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:58799 (local address 127.2.39.254:46787)
0926 06:25:49.723441 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:49.723443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:49.723512 (+ 69us) server_negotiation.cc:407] Connection header received
0926 06:25:49.723582 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:49.723583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:49.723594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:49.723642 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:49.723985 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.724492 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.725189 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:49.725367 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:49.725418 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:49.727912 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:58799: BlockingRecv error: recv got EOF from 127.2.39.195:58799 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 06:25:50 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:50 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:50.492427 11386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:50.487940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:47435)
0926 06:25:50.488075 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:50.488089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:50.488137 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:50.488284 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:50.488285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:50.488442 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:50.488485 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:50.488486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.489135 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.489135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:50.489703 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:50.489705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.490110 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.490111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:50.490147 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:50.490610 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:50.490614 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:50.492359 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
W20260926 06:25:50.492538 11387 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:50.488004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:47435 (local address 127.2.39.254:46787)
0926 06:25:50.488130 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:50.488132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:50.488184 (+ 52us) server_negotiation.cc:407] Connection header received
0926 06:25:50.488205 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:50.488205 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:50.488215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:50.488265 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:50.488562 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.489070 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.489801 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.490035 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.490107 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:50.492454 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:47435: BlockingRecv error: recv got EOF from 127.2.39.193:47435 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
Sep 26 06:25:50 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:50 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:50.601718 11388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:50.596974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:34241)
0926 06:25:50.597152 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:50.597165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:50.597259 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:50.597396 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:50.597397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:50.597640 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:50.597695 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:50.597698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.598434 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.598435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:50.598980 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:50.598981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.599362 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.599363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:50.599419 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:50.599839 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:50.599844 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:50.601611 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":113,"threads_started":1}
W20260926 06:25:50.601747 11387 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:50.597071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:34241 (local address 127.2.39.254:46787)
0926 06:25:50.597153 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:50.597155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:50.597241 (+ 86us) server_negotiation.cc:407] Connection header received
0926 06:25:50.597317 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:50.597318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:50.597331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:50.597379 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:50.597813 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.598340 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.599073 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.599287 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.599364 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:50.601684 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:34241: BlockingRecv error: recv got EOF from 127.2.39.194:34241 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 06:25:50 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:50 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:50.733263 11389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:50.728449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:34383)
0926 06:25:50.728596 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:50.728611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:50.728688 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:50.728887 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:50.728888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:50.729085 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:50.729128 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:50.729130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.729919 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.729920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:50.730448 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:50.730449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.730810 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.730811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:50.730878 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:50.731402 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:50.731406 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:50.733206 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":93,"threads_started":1}
W20260926 06:25:50.733413 11387 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:50.728559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:34383 (local address 127.2.39.254:46787)
0926 06:25:50.728626 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:50.728629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:50.728668 (+ 39us) server_negotiation.cc:407] Connection header received
0926 06:25:50.728774 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:50.728775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:50.728790 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:50.728881 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:50.729207 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.729814 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.730532 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:50.730722 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:50.730808 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:50.733297 (+ 2489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:34383: BlockingRecv error: recv got EOF from 127.2.39.195:34383 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 06:25:51 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:51 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:51.502188 11390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:51.493041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:54805)
0926 06:25:51.493217 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:51.493236 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:51.493280 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:51.493454 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:51.493454 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:51.493707 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:51.493766 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:51.493768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.494623 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.494624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:51.495170 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:51.495172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.495635 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.495636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:51.495673 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:51.500102 (+ 4429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:51.500107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:51.502119 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
W20260926 06:25:51.502394 11391 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:51.493100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:54805 (local address 127.2.39.254:46787)
0926 06:25:51.493261 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:51.493262 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:51.493287 (+ 25us) server_negotiation.cc:407] Connection header received
0926 06:25:51.493332 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:51.493332 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:51.493343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:51.493434 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:51.493864 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.494556 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.495279 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.495559 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.495630 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:51.502245 (+ 6615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:54805: BlockingRecv error: recv got EOF from 127.2.39.193:54805 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
Sep 26 06:25:51 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:51 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:51.606966 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:51.602204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:33263)
0926 06:25:51.602440 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:51.602452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:51.602540 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:51.602707 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:51.602708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:51.602890 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:51.602936 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:51.602937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.603568 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.603569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:51.604124 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:51.604125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.604441 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.604442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:51.604486 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:51.604990 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:51.604996 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:51.606895 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":98,"threads_started":1}
W20260926 06:25:51.607080 11391 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:51.602328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:33263 (local address 127.2.39.254:46787)
0926 06:25:51.602376 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:51.602378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:51.602551 (+ 173us) server_negotiation.cc:407] Connection header received
0926 06:25:51.602605 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:51.602606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:51.602619 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:51.602690 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:51.603001 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.603477 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.604195 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.604375 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.604425 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:51.606986 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:33263: BlockingRecv error: recv got EOF from 127.2.39.194:33263 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 06:25:51 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:51 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:51.738320 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:51.733767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:33227)
0926 06:25:51.733945 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:51.733962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:51.734038 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:51.734166 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:51.734167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:51.734346 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:51.734390 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:51.734391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.735058 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.735059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:51.735618 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:51.735619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.735986 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.735987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:51.736044 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:51.736475 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:51.736479 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:51.738263 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":108,"threads_started":1}
W20260926 06:25:51.738437 11391 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:51.733833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:33227 (local address 127.2.39.254:46787)
0926 06:25:51.733921 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:51.733923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:51.734023 (+ 100us) server_negotiation.cc:407] Connection header received
0926 06:25:51.734095 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:51.734096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:51.734111 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:51.734162 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:51.734479 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.734958 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.735707 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:51.735913 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:51.735988 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:51.738355 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:33227: BlockingRecv error: recv got EOF from 127.2.39.195:33227 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 06:25:52 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:52 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:52.507154 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:52.502605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:40789)
0926 06:25:52.502737 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:52.502748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:52.502807 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:52.502945 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:52.502946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:52.503104 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:52.503148 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:52.503150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.503789 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.503789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:52.504316 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:52.504318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.504642 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.504643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:52.504685 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:52.505295 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:52.505300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:52.507091 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
W20260926 06:25:52.507223 11395 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:52.502674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:40789 (local address 127.2.39.254:46787)
0926 06:25:52.502803 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:52.502805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:52.502813 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:52.502856 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:52.502856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:52.502867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:52.502926 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:52.503214 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.503718 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.504388 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.504579 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.504628 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:52.507173 (+ 2545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:40789: BlockingRecv error: recv got EOF from 127.2.39.193:40789 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
Sep 26 06:25:52 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:52 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:52.613619 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:52.607510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:49259)
0926 06:25:52.607645 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:52.607656 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:52.607719 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:52.607944 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:52.607945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:52.608177 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:52.608251 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:52.608253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.608920 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.608922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:52.609861 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:52.609864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.610287 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.610287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:52.610342 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:52.611178 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:52.611183 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:52.613500 (+ 2317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
W20260926 06:25:52.613731 11395 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:52.607549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:49259 (local address 127.2.39.254:46787)
0926 06:25:52.607596 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:52.607598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:52.607702 (+ 104us) server_negotiation.cc:407] Connection header received
0926 06:25:52.607795 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:52.607796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:52.607810 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:52.607881 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:52.608336 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.608835 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.609956 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.610210 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.610289 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:52.613653 (+ 3364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:49259: BlockingRecv error: recv got EOF from 127.2.39.194:49259 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 06:25:52 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:52 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.195@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:52.743657 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:52.738779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.195:55033)
0926 06:25:52.738963 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:52.738974 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:52.739059 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:52.739245 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:52.739246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:52.739434 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:52.739478 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:52.739479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.740152 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.740153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:52.740732 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:52.740748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.741113 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.741114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:52.741192 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:52.741691 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:52.741695 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:52.743564 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":54,"threads_started":1}
W20260926 06:25:52.743695 11395 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:52.738857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:55033 (local address 127.2.39.254:46787)
0926 06:25:52.738967 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:52.738969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:52.739046 (+ 77us) server_negotiation.cc:407] Connection header received
0926 06:25:52.739143 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:52.739144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:52.739157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:52.739228 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:52.739567 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.740067 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.740833 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:52.741028 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:52.741112 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:52.743636 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.195:55033: BlockingRecv error: recv got EOF from 127.2.39.195:55033 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 06:25:53 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:53 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.193@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:53.512221 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:53.507659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.193:36499)
0926 06:25:53.507797 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:53.507807 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:53.507856 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:53.507963 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:53.507964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:53.508129 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:53.508173 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:53.508175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.508809 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.508810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:53.509331 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:53.509332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.509800 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.509800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:53.509864 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:53.510331 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:53.510335 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:53.512163 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
W20260926 06:25:53.512356 11399 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:53.507751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:36499 (local address 127.2.39.254:46787)
0926 06:25:53.507856 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:53.507857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:53.507866 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:25:53.507891 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:53.507892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:53.507902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:53.507958 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:53.508236 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.508726 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.509443 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.509723 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.509782 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:53.512269 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.193:36499: BlockingRecv error: recv got EOF from 127.2.39.193:36499 (error 108)
Metrics: {"server-negotiator.queue_time_us":80,"thread_start_us":47,"threads_started":1}
Sep 26 06:25:53 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 06:25:53 dist-test-slave-fkbn krb5kdc[10340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.39.194@KRBTEST.COM for oryx/127.2.39.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 06:25:53.618582 11400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:53.614124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46787 (local address 127.2.39.194:40093)
0926 06:25:53.614242 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:53.614253 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:53.614316 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:53.614430 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:53.614431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:53.614587 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:53.614633 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:53.614635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.615294 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.615296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:53.615866 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:53.615869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.616197 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.616198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:53.616239 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:53.616775 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:53.616779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:53.618524 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.39.254:46787: Server oryx/127.2.39.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
W20260926 06:25:53.618717 11399 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:53.614166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:40093 (local address 127.2.39.254:46787)
0926 06:25:53.614209 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:53.614211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:53.614302 (+ 91us) server_negotiation.cc:407] Connection header received
0926 06:25:53.614358 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:53.614359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:53.614369 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:53.614417 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:53.614690 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.615197 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.615945 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.616134 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.616184 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:53.618636 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.39.194:40093: BlockingRecv error: recv got EOF from 127.2.39.194:40093 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:25:53.656852 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 10924
I20260926 06:25:53.661181 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 11076
I20260926 06:25:53.665621 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 11209
I20260926 06:25:53.669945 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 10854
2026-09-26T06:25:53Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17493 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T06:25:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:53Z Disabled control of system clock
I20260926 06:25:53.697101 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:32887
--webserver_interface=127.2.39.254
--webserver_port=0
--builtin_ntp_servers=127.2.39.212:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:32887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 06:25:53.776619 11404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:53.776821 11404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:53.776841 11404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:53.777999 11404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:53.778038 11404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:53.778050 11404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:53.778064 11404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:53.778075 11404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:53.779309 11404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:32887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:32887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:53.779495 11404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:53.779721 11404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:53.785331 11412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:53.785401 11404 server_base.cc:1034] running on GCE node
W20260926 06:25:53.785318 11410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:53.785282 11409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:53.785983 11404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:53.786240 11404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:53.787420 11404 hybrid_clock.cc:648] HybridClock initialized: now 1790403953787395 us; error 41 us; skew 500 ppm
I20260926 06:25:53.788717 11404 webserver.cc:460] Webserver started at http://127.2.39.254:35905/ using document root <none> and password file <none>
I20260926 06:25:53.788949 11404 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:53.789014 11404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:53.789170 11404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:53.790283 11404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "674161a362e2453b8fd344cb47cbbc3e"
format_stamp: "Formatted at 2026-09-26 06:25:53 on dist-test-slave-fkbn"
server_key: "0c3f1823aeab4a2eb505dc33f4782a6d"
server_key_iv: "848a1424d8be4f70860cf375ba4f0c59"
server_key_version: "encryptionkey@0"
I20260926 06:25:53.790673 11404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "674161a362e2453b8fd344cb47cbbc3e"
format_stamp: "Formatted at 2026-09-26 06:25:53 on dist-test-slave-fkbn"
server_key: "0c3f1823aeab4a2eb505dc33f4782a6d"
server_key_iv: "848a1424d8be4f70860cf375ba4f0c59"
server_key_version: "encryptionkey@0"
I20260926 06:25:53.792433 11404 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 06:25:53.793367 11418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:53.793555 11404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:53.793680 11404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "674161a362e2453b8fd344cb47cbbc3e"
format_stamp: "Formatted at 2026-09-26 06:25:53 on dist-test-slave-fkbn"
server_key: "0c3f1823aeab4a2eb505dc33f4782a6d"
server_key_iv: "848a1424d8be4f70860cf375ba4f0c59"
server_key_version: "encryptionkey@0"
I20260926 06:25:53.793748 11404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:53.804688 11404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:53.804965 11404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:53.805104 11404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:53.808773 11404 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:32887
I20260926 06:25:53.808820 11470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:32887 every 8 connection(s)
I20260926 06:25:53.809151 11404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:53.809866 11471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:53.811136 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 11404
I20260926 06:25:53.811211 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:53.811478 2207 external_mini_cluster.cc:1428] Setting key 26153209848160049f2ff619de520047
I20260926 06:25:53.812798 11471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e: Bootstrap starting.
I20260926 06:25:53.813515 11471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:53.813849 11471 log.cc:824] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:53.814563 11471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e: No bootstrap required, opened a new log
I20260926 06:25:53.816617 11471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674161a362e2453b8fd344cb47cbbc3e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 32887 } }
I20260926 06:25:53.816743 11471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:53.816764 11471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 674161a362e2453b8fd344cb47cbbc3e, State: Initialized, Role: FOLLOWER
I20260926 06:25:53.816915 11471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674161a362e2453b8fd344cb47cbbc3e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 32887 } }
I20260926 06:25:53.817030 11471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:53.817075 11471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:53.817123 11471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:53.817713 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:53.812004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.2.39.254:32887)
0926 06:25:53.812213 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:53.812223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:53.812255 (+ 32us) server_negotiation.cc:407] Connection header received
0926 06:25:53.812772 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:53.812781 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:53.812881 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:53.813024 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:53.813251 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.814600 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.815422 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.816047 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.816189 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:53.816942 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:53.816946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:53.816947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:53.817199 (+ 252us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:53.817244 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:53.817255 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:53.817324 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:53.817406 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 06:25:53.817483 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260926 06:25:53.817744 11471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674161a362e2453b8fd344cb47cbbc3e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 32887 } }
I20260926 06:25:53.817857 11471 leader_election.cc:302] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 674161a362e2453b8fd344cb47cbbc3e; no voters:
I20260926 06:25:53.818017 11471 leader_election.cc:288] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:53.818090 11476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:53.818248 11476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [term 1 LEADER]: Becoming Leader. State: Replica: 674161a362e2453b8fd344cb47cbbc3e, State: Running, Role: LEADER
I20260926 06:25:53.818280 11471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:53.818442 11476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674161a362e2453b8fd344cb47cbbc3e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 32887 } }
I20260926 06:25:53.818960 11477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 674161a362e2453b8fd344cb47cbbc3e. Latest consensus state: current_term: 1 leader_uuid: "674161a362e2453b8fd344cb47cbbc3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674161a362e2453b8fd344cb47cbbc3e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 32887 } } }
I20260926 06:25:53.818987 11478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "674161a362e2453b8fd344cb47cbbc3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "674161a362e2453b8fd344cb47cbbc3e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 32887 } } }
I20260926 06:25:53.819100 11477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:53.819103 11478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:53.819295 11485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:53.819793 11485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:53.821300 11485 catalog_manager.cc:1347] Generated new cluster ID: f986810bcc304ceca63cf158d14fc6f4
I20260926 06:25:53.821349 11485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:53.837229 11485 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:53.838008 11485 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:53.840955 11485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 674161a362e2453b8fd344cb47cbbc3e: Generated new TSK 0
I20260926 06:25:53.841130 11485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:25:53.844111 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:32887
--builtin_ntp_servers=127.2.39.212:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 06:25:53.928659 11495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:53.928856 11495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:53.928877 11495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:53.930085 11495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:53.930132 11495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:53.930158 11495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:53.931527 11495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:53.931789 11495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:53.932093 11495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:53.937244 11501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:53.937244 11500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:53.937314 11503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:53.937845 11495 server_base.cc:1034] running on GCE node
I20260926 06:25:53.938041 11495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:53.938289 11495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:53.939450 11495 hybrid_clock.cc:648] HybridClock initialized: now 1790403953939437 us; error 29 us; skew 500 ppm
I20260926 06:25:53.940845 11495 webserver.cc:460] Webserver started at http://127.2.39.193:38331/ using document root <none> and password file <none>
I20260926 06:25:53.941099 11495 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:53.941155 11495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:53.941272 11495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:53.942682 11495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "01ff30a237b642adaab6ba51d0418d10"
format_stamp: "Formatted at 2026-09-26 06:25:53 on dist-test-slave-fkbn"
server_key: "143ec421b4465e68472734152b09f798"
server_key_iv: "f9a1fe47fa484d31cc5264be02f5bf9e"
server_key_version: "encryptionkey@0"
I20260926 06:25:53.943076 11495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "01ff30a237b642adaab6ba51d0418d10"
format_stamp: "Formatted at 2026-09-26 06:25:53 on dist-test-slave-fkbn"
server_key: "143ec421b4465e68472734152b09f798"
server_key_iv: "f9a1fe47fa484d31cc5264be02f5bf9e"
server_key_version: "encryptionkey@0"
I20260926 06:25:53.944702 11495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:53.945616 11509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:53.945820 11495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:53.945914 11495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "01ff30a237b642adaab6ba51d0418d10"
format_stamp: "Formatted at 2026-09-26 06:25:53 on dist-test-slave-fkbn"
server_key: "143ec421b4465e68472734152b09f798"
server_key_iv: "f9a1fe47fa484d31cc5264be02f5bf9e"
server_key_version: "encryptionkey@0"
I20260926 06:25:53.946008 11495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:53.977458 11495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:53.977846 11495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:53.978003 11495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:53.978268 11495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:53.978617 11495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:53.978652 11495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:53.978674 11495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:53.978725 11495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:53.984560 11495 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:40357
I20260926 06:25:53.984666 11622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:40357 every 8 connection(s)
I20260926 06:25:53.985004 11495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:53.990377 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:53.985801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:41683 (local address 127.2.39.254:32887)
0926 06:25:53.985851 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:53.985853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:53.986072 (+ 219us) server_negotiation.cc:407] Connection header received
0926 06:25:53.986608 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:53.986609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:53.986621 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:53.986678 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:53.987344 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.987849 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.988851 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.989106 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.989185 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:53.990089 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:53.990090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:53.990090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:53.990095 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:53.990110 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:53.990115 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:53.990167 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:53.990296 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 06:25:53.990318 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:25:53.990657 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:53.985771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:32887 (local address 127.2.39.193:41683)
0926 06:25:53.985945 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:53.985979 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:53.986494 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:53.986753 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:53.986757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:53.986767 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:53.987250 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:53.987253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.987947 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.987948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:53.988748 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:53.988750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:53.989203 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:53.989204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:53.989361 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:53.989940 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:53.989940 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:53.989970 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:53.989991 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:53.990201 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:53.990207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:53.990210 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:53.990285 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:25:53.990419 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260926 06:25:53.991201 11623 heartbeater.cc:347] Connected to a master server at 127.2.39.254:32887
I20260926 06:25:53.991307 11623 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:53.991487 11623 heartbeater.cc:511] Master 127.2.39.254:32887 requested a full tablet report, sending...
I20260926 06:25:53.992022 11435 ts_manager.cc:194] Registered new tserver with Master: 01ff30a237b642adaab6ba51d0418d10 (127.2.39.193:40357)
I20260926 06:25:53.992544 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 11495
I20260926 06:25:53.992627 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:53.992688 11435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.193:41683
I20260926 06:25:53.992919 2207 external_mini_cluster.cc:1428] Setting key 3e14ee0b9e6c74426d0d1e3f0123ddb2
I20260926 06:25:53.993858 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:32887
--builtin_ntp_servers=127.2.39.212:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 06:25:54.073318 11626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:54.073520 11626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:54.073560 11626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:54.074744 11626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:54.074783 11626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:54.074808 11626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:54.076124 11626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:54.076375 11626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:54.076681 11626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:54.081723 11634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.081732 11632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:54.081761 11626 server_base.cc:1034] running on GCE node
W20260926 06:25:54.081758 11631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:54.082150 11626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:54.082343 11626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:54.083493 11626 hybrid_clock.cc:648] HybridClock initialized: now 1790403954083478 us; error 35 us; skew 500 ppm
I20260926 06:25:54.084925 11626 webserver.cc:460] Webserver started at http://127.2.39.194:34749/ using document root <none> and password file <none>
I20260926 06:25:54.085182 11626 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:54.085256 11626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:54.085405 11626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:54.086527 11626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "e7794f99c79d46c99d3211cea6ed6e9a"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "89bb0dbb7d97200287ca03cd0992e8fc"
server_key_iv: "09181f08de0e16cac7ccbba77fb1f717"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.086889 11626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "e7794f99c79d46c99d3211cea6ed6e9a"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "89bb0dbb7d97200287ca03cd0992e8fc"
server_key_iv: "09181f08de0e16cac7ccbba77fb1f717"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.088220 11626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:54.088999 11640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.089185 11626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:54.089279 11626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "e7794f99c79d46c99d3211cea6ed6e9a"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "89bb0dbb7d97200287ca03cd0992e8fc"
server_key_iv: "09181f08de0e16cac7ccbba77fb1f717"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.089359 11626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:54.097935 11626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:54.098230 11626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:54.098398 11626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:54.098624 11626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:54.098960 11626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:54.098997 11626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.099043 11626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:54.099072 11626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.105051 11626 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:35195
I20260926 06:25:54.105156 11753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:35195 every 8 connection(s)
I20260926 06:25:54.105474 11626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:54.108327 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 11626
I20260926 06:25:54.108444 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:54.108650 2207 external_mini_cluster.cc:1428] Setting key a391279157bd0a28ade029e723b8c2d6
I20260926 06:25:54.109685 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:32887
--builtin_ntp_servers=127.2.39.212:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 06:25:54.111164 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.106332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:33295 (local address 127.2.39.254:32887)
0926 06:25:54.106413 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:54.106415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:54.106617 (+ 202us) server_negotiation.cc:407] Connection header received
0926 06:25:54.107125 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:54.107126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:54.107136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:54.107203 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:54.107879 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:54.108379 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:54.109592 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:54.109813 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:54.109875 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:54.110827 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:54.110833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:54.110833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:54.110839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:54.110860 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:54.110864 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:54.110947 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:54.111053 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 06:25:54.111081 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 06:25:54.111438 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.106259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:32887 (local address 127.2.39.194:33295)
0926 06:25:54.106481 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:54.106522 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:54.107011 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:54.107279 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:54.107283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:54.107290 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:54.107780 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:54.107785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:54.108529 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:54.108530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:54.109468 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:54.109471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:54.109920 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:54.109922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:54.110044 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:54.110619 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:54.110619 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:54.110649 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:54.110673 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:54.110949 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:54.110955 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:54.110958 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:54.111055 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 06:25:54.111163 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260926 06:25:54.111928 11754 heartbeater.cc:347] Connected to a master server at 127.2.39.254:32887
I20260926 06:25:54.112059 11754 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:54.112265 11754 heartbeater.cc:511] Master 127.2.39.254:32887 requested a full tablet report, sending...
I20260926 06:25:54.112639 11435 ts_manager.cc:194] Registered new tserver with Master: e7794f99c79d46c99d3211cea6ed6e9a (127.2.39.194:35195)
I20260926 06:25:54.113020 11435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.194:33295
W20260926 06:25:54.194466 11757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:54.194697 11757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:54.194736 11757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:54.195958 11757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:54.196018 11757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:54.196059 11757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:54.197475 11757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:32887
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:54.197753 11757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:54.198065 11757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:54.203040 11762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.203022 11765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.203022 11763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:54.203380 11757 server_base.cc:1034] running on GCE node
I20260926 06:25:54.203605 11757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:54.203860 11757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:54.205001 11757 hybrid_clock.cc:648] HybridClock initialized: now 1790403954204990 us; error 27 us; skew 500 ppm
I20260926 06:25:54.206251 11757 webserver.cc:460] Webserver started at http://127.2.39.195:35941/ using document root <none> and password file <none>
I20260926 06:25:54.206486 11757 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:54.206565 11757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:54.206719 11757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:54.207767 11757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "5e4359624a8a438a9fa6945771cfc13b"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "d3c27508f13b59e4fab0a76b62b21f5d"
server_key_iv: "a1addeb083d82a07cdbf3387df02493f"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.208119 11757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "5e4359624a8a438a9fa6945771cfc13b"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "d3c27508f13b59e4fab0a76b62b21f5d"
server_key_iv: "a1addeb083d82a07cdbf3387df02493f"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.209384 11757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:54.210148 11771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.210325 11757 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.210414 11757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "5e4359624a8a438a9fa6945771cfc13b"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "d3c27508f13b59e4fab0a76b62b21f5d"
server_key_iv: "a1addeb083d82a07cdbf3387df02493f"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.210489 11757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:54.223872 11757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:54.224179 11757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:54.224324 11757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:54.224557 11757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:54.224895 11757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:54.224951 11757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.224992 11757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:54.225019 11757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.230835 11757 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:33463
I20260926 06:25:54.230892 11884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:33463 every 8 connection(s)
I20260926 06:25:54.231215 11757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:54.235091 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 11757
I20260926 06:25:54.235203 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:54.235419 2207 external_mini_cluster.cc:1428] Setting key f9e85f22db1173ced09a8d4148983577
I20260926 06:25:54.236457 11474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.231912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:39027 (local address 127.2.39.254:32887)
0926 06:25:54.232003 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:54.232006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:54.232215 (+ 209us) server_negotiation.cc:407] Connection header received
0926 06:25:54.232706 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:54.232708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:54.232718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:54.232773 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:54.233401 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:54.233968 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:54.234920 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:54.235111 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:54.235184 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:54.236169 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:54.236171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:54.236171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:54.236175 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:54.236190 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:54.236195 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:54.236253 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:54.236346 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 06:25:54.236372 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 06:25:54.236713 11887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.231885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:32887 (local address 127.2.39.195:39027)
0926 06:25:54.232091 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:54.232124 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:54.232612 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:54.232826 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:54.232830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:54.232842 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:54.233322 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:54.233326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:54.234045 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:54.234045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:54.234825 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:54.234826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:54.235196 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:54.235197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:54.235368 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:54.236011 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:54.236011 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:54.236050 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:54.236078 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:54.236259 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:54.236263 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:54.236266 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:54.236333 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:25:54.236460 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":104,"threads_started":1}
I20260926 06:25:54.237193 11885 heartbeater.cc:347] Connected to a master server at 127.2.39.254:32887
I20260926 06:25:54.237282 11885 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:54.237424 11885 heartbeater.cc:511] Master 127.2.39.254:32887 requested a full tablet report, sending...
I20260926 06:25:54.237798 11435 ts_manager.cc:194] Registered new tserver with Master: 5e4359624a8a438a9fa6945771cfc13b (127.2.39.195:33463)
I20260926 06:25:54.238165 11435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.195:39027
I20260926 06:25:54.247282 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 06:25:54.249104 11474 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:54.248730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.2.39.254:32887)
0926 06:25:54.248776 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:54.248777 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:54.248827 (+ 50us) server_negotiation.cc:407] Connection header received
0926 06:25:54.248890 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:54.248891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:54.248898 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:54.248938 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:54.249035 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50786: BlockingRecv error: recv got EOF from 127.0.0.1:50786 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:25:54.249212 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 11495
I20260926 06:25:54.253912 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 11626
I20260926 06:25:54.258412 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 11757
I20260926 06:25:54.262733 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 11404
2026-09-26T06:25:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (591 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T06:25:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:54Z Disabled control of system clock
I20260926 06:25:54.289106 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36861
--webserver_interface=127.2.39.254
--webserver_port=0
--builtin_ntp_servers=127.2.39.212:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:36861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 06:25:54.371582 11896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:54.371831 11896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:54.371869 11896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:54.373504 11896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:54.373569 11896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:54.373615 11896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:54.373647 11896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:54.373675 11896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:54.375026 11896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:36861
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:54.375312 11896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:54.375617 11896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:54.380442 11901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.380482 11904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.380522 11902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:54.380527 11896 server_base.cc:1034] running on GCE node
I20260926 06:25:54.380882 11896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:54.381162 11896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:54.382321 11896 hybrid_clock.cc:648] HybridClock initialized: now 1790403954382306 us; error 28 us; skew 500 ppm
I20260926 06:25:54.383601 11896 webserver.cc:460] Webserver started at http://127.2.39.254:42963/ using document root <none> and password file <none>
I20260926 06:25:54.383857 11896 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:54.383927 11896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:54.384081 11896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:54.385146 11896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "52d02224986548e7980b4f604cb396da"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "809a3b2460cf8914ff2924f846deb7ed"
server_key_iv: "b98033fd2fa80d3709b880716e666748"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.385495 11896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "52d02224986548e7980b4f604cb396da"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "809a3b2460cf8914ff2924f846deb7ed"
server_key_iv: "b98033fd2fa80d3709b880716e666748"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.386780 11896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:54.387470 11910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.387637 11896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:54.387738 11896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "52d02224986548e7980b4f604cb396da"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "809a3b2460cf8914ff2924f846deb7ed"
server_key_iv: "b98033fd2fa80d3709b880716e666748"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.387811 11896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:54.391189 11896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:54.391439 11896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:54.391589 11896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:54.395408 11896 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:36861
I20260926 06:25:54.395454 11962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:36861 every 8 connection(s)
I20260926 06:25:54.395812 11896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:54.396538 11963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:54.399608 11963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da: Bootstrap starting.
I20260926 06:25:54.400257 11963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:54.400609 11963 log.cc:824] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:54.401347 11963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da: No bootstrap required, opened a new log
I20260926 06:25:54.403103 11963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d02224986548e7980b4f604cb396da" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36861 } }
I20260926 06:25:54.403252 11963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:54.403275 11963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52d02224986548e7980b4f604cb396da, State: Initialized, Role: FOLLOWER
I20260926 06:25:54.403463 11963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d02224986548e7980b4f604cb396da" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36861 } }
I20260926 06:25:54.403550 11963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:54.403585 11963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:54.403656 11963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:54.403643 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 11896
I20260926 06:25:54.403731 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:54.403915 2207 external_mini_cluster.cc:1428] Setting key aab0110e4ae5a33ed5030ed26cf49dc7
I20260926 06:25:54.404323 11963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d02224986548e7980b4f604cb396da" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36861 } }
I20260926 06:25:54.404438 11963 leader_election.cc:302] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52d02224986548e7980b4f604cb396da; no voters:
I20260926 06:25:54.404618 11963 leader_election.cc:288] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:54.404640 11968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:54.404876 11963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:54.404889 11968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [term 1 LEADER]: Becoming Leader. State: Replica: 52d02224986548e7980b4f604cb396da, State: Running, Role: LEADER
I20260926 06:25:54.405174 11968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d02224986548e7980b4f604cb396da" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36861 } }
I20260926 06:25:54.406172 11969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52d02224986548e7980b4f604cb396da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d02224986548e7980b4f604cb396da" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36861 } } }
I20260926 06:25:54.406241 11970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52d02224986548e7980b4f604cb396da. Latest consensus state: current_term: 1 leader_uuid: "52d02224986548e7980b4f604cb396da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52d02224986548e7980b4f604cb396da" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36861 } } }
I20260926 06:25:54.406172 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.404449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.2.39.254:36861)
0926 06:25:54.404673 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:54.404683 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:54.404721 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:25:54.404848 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:54.404856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:54.404905 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:54.405059 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:54.405763 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:54.405766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:54.405766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:54.405769 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:54.405803 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:54.405815 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:54.405880 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:54.405958 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 06:25:54.406013 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260926 06:25:54.406299 11970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:54.406296 11969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:54.406577 11977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:54.407064 11977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:54.410686 11977 catalog_manager.cc:1347] Generated new cluster ID: 8eaefdde33fe4ff58ef4b43070c0f819
I20260926 06:25:54.410764 11977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:54.442655 11977 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:54.442862 11977 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:54.447574 11977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52d02224986548e7980b4f604cb396da: Generated new TSK 0
I20260926 06:25:54.447767 11977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:25:54.458526 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36861
--builtin_ntp_servers=127.2.39.212:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 06:25:54.538734 11987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:54.538942 11987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:54.538964 11987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:54.540134 11987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:54.540176 11987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:54.540201 11987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:54.541518 11987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:54.541785 11987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:54.542101 11987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:54.547336 11993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.547358 11992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.547346 11995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:54.547976 11987 server_base.cc:1034] running on GCE node
I20260926 06:25:54.548237 11987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:54.548532 11987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:54.549688 11987 hybrid_clock.cc:648] HybridClock initialized: now 1790403954549672 us; error 26 us; skew 500 ppm
I20260926 06:25:54.550957 11987 webserver.cc:460] Webserver started at http://127.2.39.193:43863/ using document root <none> and password file <none>
I20260926 06:25:54.551245 11987 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:54.551313 11987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:54.551427 11987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:54.552513 11987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "3fe0f411f875436092f98b825af6d5c1"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "6b19fcb2d93b4925d796890831df9d76"
server_key_iv: "2d8c32a9828db476f4ee92f160e8de6d"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.552811 11987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "3fe0f411f875436092f98b825af6d5c1"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "6b19fcb2d93b4925d796890831df9d76"
server_key_iv: "2d8c32a9828db476f4ee92f160e8de6d"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.554139 11987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:54.554783 12001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.554965 11987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:54.555035 11987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "3fe0f411f875436092f98b825af6d5c1"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "6b19fcb2d93b4925d796890831df9d76"
server_key_iv: "2d8c32a9828db476f4ee92f160e8de6d"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.555140 11987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:54.558498 11987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:54.558758 11987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:54.558890 11987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:54.559113 11987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:54.559419 11987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:54.559458 11987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.559514 11987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:54.559543 11987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.565191 11987 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:35597
I20260926 06:25:54.565263 12114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:35597 every 8 connection(s)
I20260926 06:25:54.565636 11987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:54.568686 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.566578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:48465 (local address 127.2.39.254:36861)
0926 06:25:54.566656 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:54.566658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:54.566759 (+ 101us) server_negotiation.cc:407] Connection header received
0926 06:25:54.567221 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:54.567222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:54.567228 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:54.567281 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:54.568499 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:54.568500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:54.568500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:54.568504 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:54.568519 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:54.568523 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:54.568575 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:54.568621 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 06:25:54.568633 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 06:25:54.568885 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.566404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36861 (local address 127.2.39.193:48465)
0926 06:25:54.566637 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:54.566690 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:54.567141 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:54.567334 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:54.567337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:54.567346 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:54.568349 (+ 1003us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:54.568349 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:54.568377 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:54.568405 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:54.568572 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:54.568575 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:54.568577 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:54.568627 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:25:54.568674 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260926 06:25:54.569325 12115 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36861
I20260926 06:25:54.569404 12115 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:54.569592 12115 heartbeater.cc:511] Master 127.2.39.254:36861 requested a full tablet report, sending...
I20260926 06:25:54.570117 11927 ts_manager.cc:194] Registered new tserver with Master: 3fe0f411f875436092f98b825af6d5c1 (127.2.39.193:35597)
I20260926 06:25:54.573669 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 11987
I20260926 06:25:54.573781 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:54.573976 2207 external_mini_cluster.cc:1428] Setting key 4133d698f311630ffdbca3221bf5b75c
I20260926 06:25:54.574865 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36861
--builtin_ntp_servers=127.2.39.212:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 06:25:54.654351 12118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:54.654547 12118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:54.654569 12118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:54.655733 12118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:54.655772 12118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:54.655826 12118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:54.657132 12118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:54.657379 12118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:54.657713 12118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:25:54.662701 12118 server_base.cc:1034] running on GCE node
W20260926 06:25:54.662684 12123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.662684 12126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.662647 12124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:54.663163 12118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:54.663401 12118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:54.664541 12118 hybrid_clock.cc:648] HybridClock initialized: now 1790403954664529 us; error 27 us; skew 500 ppm
I20260926 06:25:54.665829 12118 webserver.cc:460] Webserver started at http://127.2.39.194:44249/ using document root <none> and password file <none>
I20260926 06:25:54.666077 12118 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:54.666123 12118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:54.666280 12118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:54.667346 12118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "2a70c558fc814502b1260741b92663c3"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "9700a9927c66f81732d61b194e4e1407"
server_key_iv: "e2a117a58a938ca6f65c8f3cbb4e5f6e"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.667689 12118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "2a70c558fc814502b1260741b92663c3"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "9700a9927c66f81732d61b194e4e1407"
server_key_iv: "e2a117a58a938ca6f65c8f3cbb4e5f6e"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.668985 12118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:25:54.669725 12132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.669898 12118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:54.669966 12118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "2a70c558fc814502b1260741b92663c3"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "9700a9927c66f81732d61b194e4e1407"
server_key_iv: "e2a117a58a938ca6f65c8f3cbb4e5f6e"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.670047 12118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:54.673298 12118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:54.673552 12118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:54.673717 12118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:54.673918 12118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:54.674204 12118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:54.674237 12118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.674255 12118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:54.674300 12118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.679994 12118 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:37381
I20260926 06:25:54.680039 12245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:37381 every 8 connection(s)
I20260926 06:25:54.680335 12118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:25:54.683312 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.681043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:52459 (local address 127.2.39.254:36861)
0926 06:25:54.681116 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:54.681118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:54.681315 (+ 197us) server_negotiation.cc:407] Connection header received
0926 06:25:54.681843 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:54.681844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:54.681852 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:54.681909 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:54.683022 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:54.683024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:54.683025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:54.683037 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:54.683055 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:54.683060 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:54.683137 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:54.683229 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 06:25:54.683244 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:25:54.683523 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.680991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36861 (local address 127.2.39.194:52459)
0926 06:25:54.681195 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:54.681235 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:54.681738 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:54.681980 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:54.681984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:54.681993 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:54.682872 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:54.682872 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:54.682904 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:54.682929 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:54.683136 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:54.683142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:54.683144 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:25:54.683240 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 06:25:54.683298 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":117,"threads_started":1}
I20260926 06:25:54.684062 12246 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36861
I20260926 06:25:54.684134 12246 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:54.684294 12246 heartbeater.cc:511] Master 127.2.39.254:36861 requested a full tablet report, sending...
I20260926 06:25:54.684708 11927 ts_manager.cc:194] Registered new tserver with Master: 2a70c558fc814502b1260741b92663c3 (127.2.39.194:37381)
I20260926 06:25:54.689484 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 12118
I20260926 06:25:54.689617 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:54.689813 2207 external_mini_cluster.cc:1428] Setting key bd2a83b8564cd23d18fc313364643e2d
I20260926 06:25:54.690758 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36861
--builtin_ntp_servers=127.2.39.212:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 06:25:54.772917 12249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:54.773159 12249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:54.773197 12249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:54.774461 12249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:54.774528 12249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:54.774571 12249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:54.775916 12249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:36861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:54.776198 12249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:54.776507 12249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:54.781721 12254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.781750 12257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:54.781749 12255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:54.781754 12249 server_base.cc:1034] running on GCE node
I20260926 06:25:54.782022 12249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:54.782219 12249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:54.783370 12249 hybrid_clock.cc:648] HybridClock initialized: now 1790403954783343 us; error 36 us; skew 500 ppm
I20260926 06:25:54.784557 12249 webserver.cc:460] Webserver started at http://127.2.39.195:36641/ using document root <none> and password file <none>
I20260926 06:25:54.784804 12249 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:54.784885 12249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:54.785033 12249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:54.786293 12249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "4f7178c6a2b7412d8fa5dc2835d2acf9"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "8f2e81358a5e1c171a6e1aa1888a6cd8"
server_key_iv: "e8b7133dcced6c752a5e9a00c7c46852"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.786736 12249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "4f7178c6a2b7412d8fa5dc2835d2acf9"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "8f2e81358a5e1c171a6e1aa1888a6cd8"
server_key_iv: "e8b7133dcced6c752a5e9a00c7c46852"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.788136 12249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:54.788854 12263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.789032 12249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:25:54.789139 12249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "4f7178c6a2b7412d8fa5dc2835d2acf9"
format_stamp: "Formatted at 2026-09-26 06:25:54 on dist-test-slave-fkbn"
server_key: "8f2e81358a5e1c171a6e1aa1888a6cd8"
server_key_iv: "e8b7133dcced6c752a5e9a00c7c46852"
server_key_version: "encryptionkey@0"
I20260926 06:25:54.789234 12249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:54.792912 12249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:54.793175 12249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:54.793322 12249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:54.793537 12249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:54.793880 12249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:54.793946 12249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.793991 12249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:54.794036 12249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:54.800787 12249 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:42707
I20260926 06:25:54.800819 12376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:42707 every 8 connection(s)
I20260926 06:25:54.801151 12249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:25:54.803920 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.801891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:51407 (local address 127.2.39.254:36861)
0926 06:25:54.801933 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:54.801935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:54.802173 (+ 238us) server_negotiation.cc:407] Connection header received
0926 06:25:54.802604 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:54.802605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:54.802609 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:54.802669 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:54.803745 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:54.803746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:54.803746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:25:54.803750 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:54.803763 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:25:54.803767 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:54.803813 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:54.803857 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 06:25:54.803868 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:25:54.804163 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:54.801862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:36861 (local address 127.2.39.195:51407)
0926 06:25:54.802044 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:54.802084 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:54.802528 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:54.802718 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:54.802721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:54.802729 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:54.803604 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:25:54.803605 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:54.803631 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:25:54.803659 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:54.803810 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:54.803812 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:54.803815 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:25:54.803864 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:25:54.803909 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260926 06:25:54.804705 12377 heartbeater.cc:347] Connected to a master server at 127.2.39.254:36861
I20260926 06:25:54.804787 12377 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:54.804888 12377 heartbeater.cc:511] Master 127.2.39.254:36861 requested a full tablet report, sending...
I20260926 06:25:54.805259 11927 ts_manager.cc:194] Registered new tserver with Master: 4f7178c6a2b7412d8fa5dc2835d2acf9 (127.2.39.195:42707)
I20260926 06:25:54.805408 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 12249
I20260926 06:25:54.805522 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:54.805733 2207 external_mini_cluster.cc:1428] Setting key a504ab1fa074363d3044308ba2a046f2
I20260926 06:25:54.806639 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 06:25:54.808578 11967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:25:54.808007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.2.39.254:36861)
0926 06:25:54.808059 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:54.808061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:54.808203 (+ 142us) server_negotiation.cc:407] Connection header received
0926 06:25:54.808277 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:54.808278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:54.808282 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:54.808331 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:54.808518 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56178: BlockingRecv error: recv got EOF from 127.0.0.1:56178 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:25:54.808579 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 11987
I20260926 06:25:54.813385 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 12118
I20260926 06:25:54.817698 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 12249
I20260926 06:25:54.821926 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 11896
2026-09-26T06:25:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (559 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:54 dist-test-slave-fkbn krb5kdc[12387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:54 dist-test-slave-fkbn krb5kdc[12387](info): set up 2 sockets
Sep 26 06:25:54 dist-test-slave-fkbn krb5kdc[12387](info): commencing operation
krb5kdc: starting...
W20260926 06:25:55.862501 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:55 dist-test-slave-fkbn krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:25:55.985064 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38169
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:38169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:56.063836 12400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:56.064029 12400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:56.064051 12400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:56.065189 12400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:25:56.065230 12400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:56.065244 12400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:56.065256 12400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:25:56.065269 12400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:25:56.066744 12400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:38169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:56.066998 12400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:56.067232 12400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:56.071940 12406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:56.072021 12400 server_base.cc:1034] running on GCE node
W20260926 06:25:56.071941 12405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:56.071969 12408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:56.072470 12400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:56.072722 12400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:56.073923 12400 hybrid_clock.cc:648] HybridClock initialized: now 1790403956073895 us; error 44 us; skew 500 ppm
Sep 26 06:25:56 dist-test-slave-fkbn krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:56.076532 12400 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:25:56.076898 12400 webserver.cc:460] Webserver started at http://127.2.39.254:34289/ using document root <none> and password file <none>
I20260926 06:25:56.077082 12400 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:56.077121 12400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:56.077203 12400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:56.078258 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "07e3fbf08581457ca085ad600c0ea16e"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "69a7bd6f069ccedf1241589e3176c051"
server_key_iv: "000f9ae30f08d0c42e3ce6e7e2f0b2c1"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.078572 12400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "07e3fbf08581457ca085ad600c0ea16e"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "69a7bd6f069ccedf1241589e3176c051"
server_key_iv: "000f9ae30f08d0c42e3ce6e7e2f0b2c1"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.079813 12400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:56.080574 12415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:56.080777 12400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:25:56.080864 12400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "07e3fbf08581457ca085ad600c0ea16e"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "69a7bd6f069ccedf1241589e3176c051"
server_key_iv: "000f9ae30f08d0c42e3ce6e7e2f0b2c1"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.080919 12400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:56.091825 12400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:56.093962 12400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:56.094074 12400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:56.097821 12400 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:38169
I20260926 06:25:56.097877 12467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:38169 every 8 connection(s)
I20260926 06:25:56.098193 12400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:25:56.099246 12468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:56.099468 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 12400
I20260926 06:25:56.099548 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:25:56.099692 2207 external_mini_cluster.cc:1428] Setting key 438d97452cb6e4f5386b72b41b5cea7b
I20260926 06:25:56.103021 12468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e: Bootstrap starting.
I20260926 06:25:56.103714 12468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:56.104023 12468 log.cc:824] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:56.104807 12468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e: No bootstrap required, opened a new log
Sep 26 06:25:56 dist-test-slave-fkbn krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:56.107306 12468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e3fbf08581457ca085ad600c0ea16e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38169 } }
I20260926 06:25:56.107510 12468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:56.107542 12468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e3fbf08581457ca085ad600c0ea16e, State: Initialized, Role: FOLLOWER
I20260926 06:25:56.107698 12468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e3fbf08581457ca085ad600c0ea16e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38169 } }
I20260926 06:25:56.107834 12468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:25:56.107875 12468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:25:56.107929 12468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:56.108580 12468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e3fbf08581457ca085ad600c0ea16e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38169 } }
I20260926 06:25:56.108716 12468 leader_election.cc:302] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07e3fbf08581457ca085ad600c0ea16e; no voters:
I20260926 06:25:56.108918 12468 leader_election.cc:288] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:25:56.108951 12473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:56.109067 12473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [term 1 LEADER]: Becoming Leader. State: Replica: 07e3fbf08581457ca085ad600c0ea16e, State: Running, Role: LEADER
I20260926 06:25:56.109154 12473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e3fbf08581457ca085ad600c0ea16e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38169 } }
I20260926 06:25:56.109215 12468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:25:56.109480 12475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07e3fbf08581457ca085ad600c0ea16e. Latest consensus state: current_term: 1 leader_uuid: "07e3fbf08581457ca085ad600c0ea16e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e3fbf08581457ca085ad600c0ea16e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38169 } } }
I20260926 06:25:56.109603 12475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:56.109753 12474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07e3fbf08581457ca085ad600c0ea16e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e3fbf08581457ca085ad600c0ea16e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38169 } } }
I20260926 06:25:56.109819 12474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e [sys.catalog]: This master's current role is: LEADER
I20260926 06:25:56.109905 12477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:25:56.110512 12477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:25:56.111726 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.100232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.2.39.254:38169)
0926 06:25:56.100427 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.100438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.100473 (+ 35us) server_negotiation.cc:407] Connection header received
0926 06:25:56.101074 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.101083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.101149 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.101304 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:56.101939 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.103302 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.104097 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.104658 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.104771 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.107463 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:56.107467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:56.107467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:56.107477 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:56.109676 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:56.110109 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:56.110111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:56.110112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:56.110215 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:56.110440 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:56.110440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:56.110441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:56.110759 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:56.111323 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.111390 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.111520 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260926 06:25:56.114647 12477 catalog_manager.cc:1347] Generated new cluster ID: 2db0dd8cb39a44b5a6ef1db3aab10b24
I20260926 06:25:56.114711 12477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:25:56.150697 12477 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:25:56.151317 12477 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:25:56.160512 12477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e: Generated new TSK 0
I20260926 06:25:56.160738 12477 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:25:56.188617 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38169
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:56.269219 12494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:56.269421 12494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:56.269444 12494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:56.270540 12494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:56.270576 12494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:56.270601 12494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:25:56.271951 12494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:56.272128 12494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:56.272295 12494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:56.277194 12499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:56.277185 12500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:56.277185 12502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:56.277604 12494 server_base.cc:1034] running on GCE node
I20260926 06:25:56.277787 12494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:56.277982 12494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:56.279111 12494 hybrid_clock.cc:648] HybridClock initialized: now 1790403956279097 us; error 28 us; skew 500 ppm
Sep 26 06:25:56 dist-test-slave-fkbn krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:56.281538 12494 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:25:56.281975 12494 webserver.cc:460] Webserver started at http://127.2.39.193:46355/ using document root <none> and password file <none>
I20260926 06:25:56.282173 12494 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:56.282214 12494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:56.282300 12494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:56.283401 12494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "4a5068b39df14635be07560c05dc1a13"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "6b4882688958860ffc5b89bf4b80be00"
server_key_iv: "0042b8a8928b621a600609d8591c8a38"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.283723 12494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "4a5068b39df14635be07560c05dc1a13"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "6b4882688958860ffc5b89bf4b80be00"
server_key_iv: "0042b8a8928b621a600609d8591c8a38"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.284977 12494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:25:56.285709 12509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:56.285871 12494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:25:56.285950 12494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "4a5068b39df14635be07560c05dc1a13"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "6b4882688958860ffc5b89bf4b80be00"
server_key_iv: "0042b8a8928b621a600609d8591c8a38"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.286002 12494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:56.293260 12494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:56.294955 12494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:56.295063 12494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:56.295241 12494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:56.295472 12494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:56.295500 12494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:56.295521 12494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:56.295542 12494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:56.300912 12494 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:33995
I20260926 06:25:56.300980 12622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:33995 every 8 connection(s)
I20260926 06:25:56.301280 12494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:25:56.303108 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 12494
I20260926 06:25:56.303186 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:25:56.303375 2207 external_mini_cluster.cc:1428] Setting key 4162a842a372ac25d671a39561aa942a
Sep 26 06:25:56 dist-test-slave-fkbn krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:56.311275 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.302003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:49327 (local address 127.2.39.254:38169)
0926 06:25:56.302050 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.302052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.302274 (+ 222us) server_negotiation.cc:407] Connection header received
0926 06:25:56.302729 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.302730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.302741 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.302793 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:56.303870 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.304632 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.305720 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.305925 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.306009 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.308391 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:56.308392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:56.308393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:56.308399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:56.309842 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:56.310299 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:56.310300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:56.310300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:56.310386 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:56.310684 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:56.310685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:56.310686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:56.310783 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:56.310835 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.311147 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.311212 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:25:56.311558 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.301987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38169 (local address 127.2.39.193:49327)
0926 06:25:56.302158 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:56.302192 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:56.302652 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:56.302841 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:56.302844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:56.303511 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:56.303790 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.303796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.304720 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.304721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.305559 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.305561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.306017 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.306018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.306146 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.306634 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:56.306642 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:56.308300 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:56.309921 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:56.309927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:56.309930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:56.310224 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:56.310462 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:56.310463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:56.310463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:56.310618 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:56.310850 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:56.310855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:56.311015 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 06:25:56.311134 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:25:56.311296 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260926 06:25:56.312114 12623 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38169
I20260926 06:25:56.312198 12623 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
I20260926 06:25:56.312390 12623 heartbeater.cc:511] Master 127.2.39.254:38169 requested a full tablet report, sending...
I20260926 06:25:56.312884 12432 ts_manager.cc:194] Registered new tserver with Master: 4a5068b39df14635be07560c05dc1a13 (127.2.39.193:33995)
I20260926 06:25:56.313540 12432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:49327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:25:56.330675 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38169
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:56.411002 12628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:56.411216 12628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:56.411249 12628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:56.412348 12628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:56.412417 12628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:56.412460 12628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:25:56.413893 12628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:56.414148 12628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:56.414376 12628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:56.419400 12636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:56.419453 12633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:56.419456 12634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:56.419780 12628 server_base.cc:1034] running on GCE node
I20260926 06:25:56.419973 12628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:56.420157 12628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:56.421293 12628 hybrid_clock.cc:648] HybridClock initialized: now 1790403956421282 us; error 28 us; skew 500 ppm
Sep 26 06:25:56 dist-test-slave-fkbn krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:56.423830 12628 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:25:56.424265 12628 webserver.cc:460] Webserver started at http://127.2.39.194:38619/ using document root <none> and password file <none>
I20260926 06:25:56.424459 12628 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:56.424500 12628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:56.424587 12628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:56.425545 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "946fa748f3f04df08466693dfd0529d5"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "fd4256de545280e3a6b76a12bee0ac84"
server_key_iv: "c857a52c2384819557753f47cf163507"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.425913 12628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "946fa748f3f04df08466693dfd0529d5"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "fd4256de545280e3a6b76a12bee0ac84"
server_key_iv: "c857a52c2384819557753f47cf163507"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.427174 12628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:56.427837 12643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:56.428098 12628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:25:56.428191 12628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "946fa748f3f04df08466693dfd0529d5"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "fd4256de545280e3a6b76a12bee0ac84"
server_key_iv: "c857a52c2384819557753f47cf163507"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.428272 12628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:56.447117 12628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:56.449474 12628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:56.449699 12628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:56.449939 12628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:56.450263 12628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:56.450317 12628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:56.450351 12628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:56.450378 12628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:56.456382 12628 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:33287
I20260926 06:25:56.456420 12756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:33287 every 8 connection(s)
I20260926 06:25:56.456755 12628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:25:56 dist-test-slave-fkbn krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:56.465543 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 12628
I20260926 06:25:56.465683 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:25:56.465889 2207 external_mini_cluster.cc:1428] Setting key d7687cf47e78aac98c9d403894ca86ae
I20260926 06:25:56.466279 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.457511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:40661 (local address 127.2.39.254:38169)
0926 06:25:56.457630 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.457633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.457805 (+ 172us) server_negotiation.cc:407] Connection header received
0926 06:25:56.458336 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.458337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.458349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.458406 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:56.459350 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.459853 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.460867 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.461082 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.461169 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.463387 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:56.463388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:56.463389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:56.463397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:56.464663 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:56.465095 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:56.465096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:56.465096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:56.465159 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:56.465450 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:56.465451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:56.465452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:56.465673 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:56.465760 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.466078 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.466181 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:25:56.466480 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.457485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38169 (local address 127.2.39.194:40661)
0926 06:25:56.457685 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:56.457722 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:56.458231 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:56.458492 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:56.458495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:56.459058 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:56.459249 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.459254 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.459949 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.459951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.460761 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.460763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.461178 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.461179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.461304 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.461828 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:56.461837 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:56.463289 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:56.464761 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:56.464768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:56.464770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:56.464994 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:56.465228 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:56.465229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:56.465229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:56.465333 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:56.465749 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:56.465756 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:56.465902 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 06:25:56.466030 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 06:25:56.466255 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":106,"threads_started":1}
I20260926 06:25:56.466897 12757 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38169
I20260926 06:25:56.467017 12757 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:56.467222 12757 heartbeater.cc:511] Master 127.2.39.254:38169 requested a full tablet report, sending...
I20260926 06:25:56.467608 12432 ts_manager.cc:194] Registered new tserver with Master: 946fa748f3f04df08466693dfd0529d5 (127.2.39.194:33287)
I20260926 06:25:56.468000 12432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:40661
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:25:56.492836 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38169
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:25:56.572494 12762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:25:56.572690 12762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:25:56.572710 12762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:25:56.573865 12762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:25:56.573915 12762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:25:56.573940 12762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:25:56.575320 12762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:25:56.575489 12762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:25:56.575656 12762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:25:56.580642 12767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:25:56.580647 12768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:56.580847 12762 server_base.cc:1034] running on GCE node
W20260926 06:25:56.580668 12770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:25:56.581113 12762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:25:56.581350 12762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:25:56.582507 12762 hybrid_clock.cc:648] HybridClock initialized: now 1790403956582455 us; error 70 us; skew 500 ppm
Sep 26 06:25:56 dist-test-slave-fkbn krb5kdc[12387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:25:56.588881 12762 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:25:56.589362 12762 webserver.cc:460] Webserver started at http://127.2.39.195:42287/ using document root <none> and password file <none>
I20260926 06:25:56.589656 12762 fs_manager.cc:359] Metadata directory not provided
I20260926 06:25:56.589725 12762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:25:56.589895 12762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:25:56.591053 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "c7454c1825184194a384344a62223a15"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "b98a144a21b23711c5352d42c3f7a97f"
server_key_iv: "552bbcd6026b9884d824702a728eee35"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.591408 12762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "c7454c1825184194a384344a62223a15"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "b98a144a21b23711c5352d42c3f7a97f"
server_key_iv: "552bbcd6026b9884d824702a728eee35"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.592773 12762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:25:56.593477 12777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:56.593673 12762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:25:56.593771 12762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "c7454c1825184194a384344a62223a15"
format_stamp: "Formatted at 2026-09-26 06:25:56 on dist-test-slave-fkbn"
server_key: "b98a144a21b23711c5352d42c3f7a97f"
server_key_iv: "552bbcd6026b9884d824702a728eee35"
server_key_version: "encryptionkey@0"
I20260926 06:25:56.593856 12762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:25:56.610616 12762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:25:56.612915 12762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:25:56.613085 12762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:25:56.613315 12762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:25:56.613695 12762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:25:56.613749 12762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:56.613792 12762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:25:56.613818 12762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:25:56.619824 12762 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:34721
I20260926 06:25:56.619864 12890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:34721 every 8 connection(s)
I20260926 06:25:56.620177 12762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:25:56 dist-test-slave-fkbn krb5kdc[12387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403956, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:25:56.627489 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 12762
I20260926 06:25:56.627579 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:25:56.627760 2207 external_mini_cluster.cc:1428] Setting key 93a03e600b981d3bef1f0768e9dd8355
I20260926 06:25:56.629281 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.620887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:33625 (local address 127.2.39.254:38169)
0926 06:25:56.620953 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.620955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.621152 (+ 197us) server_negotiation.cc:407] Connection header received
0926 06:25:56.621751 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.621752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.621762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.621816 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:56.622695 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.623176 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.624178 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.624351 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.624401 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.626502 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:56.626503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:56.626503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:56.626510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:56.627763 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:56.628332 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:56.628333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:56.628334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:56.628402 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:56.628666 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:56.628667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:56.628667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:56.628811 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:56.628864 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.629138 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.629219 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:25:56.629336 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.620870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38169 (local address 127.2.39.195:33625)
0926 06:25:56.621031 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:56.621066 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:56.621664 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:56.621871 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:56.621874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:56.622432 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:25:56.622622 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.622626 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.623258 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.623259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.624106 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.624108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.624463 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.624464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.624558 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.624992 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:25:56.625000 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:25:56.626396 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:25:56.627888 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:56.627894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:56.627897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:56.628244 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:56.628476 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:25:56.628477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:25:56.628477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:25:56.628575 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:25:56.628869 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:25:56.628873 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:25:56.628981 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 06:25:56.629072 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 06:25:56.629182 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260926 06:25:56.629909 12891 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38169
I20260926 06:25:56.629994 12891 heartbeater.cc:464] Registering TS with master...
I20260926 06:25:56.630129 12891 heartbeater.cc:511] Master 127.2.39.254:38169 requested a full tablet report, sending...
I20260926 06:25:56.630491 12432 ts_manager.cc:194] Registered new tserver with Master: c7454c1825184194a384344a62223a15 (127.2.39.195:34721)
I20260926 06:25:56.630854 12432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:33625
I20260926 06:25:56.639671 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:25:56.647075 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.640905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37848 (local address 127.2.39.254:38169)
0926 06:25:56.640971 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.640973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.641042 (+ 69us) server_negotiation.cc:407] Connection header received
0926 06:25:56.641081 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.641082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.641089 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.641137 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:25:56.641888 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.642344 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.643089 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.643274 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.643327 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.644605 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:25:56.644606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:25:56.644607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:25:56.644613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:25:56.645911 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:56.646206 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:56.646207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:56.646207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:56.646254 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:25:56.646540 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:25:56.646541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:25:56.646541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:25:56.646691 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:25:56.646772 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.646914 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.646973 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 06:25:56.648795 12432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:25:56.650494 12432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:25:56.656988 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.652650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:34721 (local address 127.0.0.1:40284)
0926 06:25:56.652764 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:56.652801 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:56.652866 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:56.653382 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:56.653382 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:56.653384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:56.653463 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.653465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.654797 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.654798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.655958 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.655960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.656680 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.656681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.656742 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.656755 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 06:25:56.656803 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:25:56.656855 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":39,"thread_start_us":69,"threads_started":1}
I20260926 06:25:56.656978 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.653028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.2.39.194:33287)
0926 06:25:56.653210 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.653214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.653222 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:56.653277 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.653279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.653294 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.653371 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:56.653598 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.654697 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.655759 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.656559 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.656635 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.656691 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.656848 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.656901 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
I20260926 06:25:56.657115 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.652796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.2.39.195:34721)
0926 06:25:56.653222 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.653226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.653234 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:25:56.653273 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.653274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.653289 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.653535 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:56.653618 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.654687 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.656043 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.656570 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.656902 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.656967 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.656989 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.657035 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":84,"threads_started":1}
I20260926 06:25:56.657308 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.652863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49548 (local address 127.2.39.193:33995)
0926 06:25:56.653018 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.653023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.653119 (+ 96us) server_negotiation.cc:407] Connection header received
0926 06:25:56.653168 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.653171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.653188 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.653271 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:56.653473 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.654337 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.655836 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.656559 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.656635 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.656691 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.657203 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.657255 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260926 06:25:56.657140 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.652690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:33995 (local address 127.0.0.1:49548)
0926 06:25:56.653026 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:56.653036 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:56.653107 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:56.653298 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:56.653300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:56.653302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:56.653371 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.653372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.654501 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.654502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.655744 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.655747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.656933 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.656934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.656990 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.656994 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:56.657042 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:25:56.657075 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":46,"threads_started":1}
I20260926 06:25:56.657691 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.652857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:33287 (local address 127.0.0.1:38012)
0926 06:25:56.653042 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:56.653051 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:56.653096 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:56.653379 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:56.653380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:56.653381 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:56.653464 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.653465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.654813 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.654814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.655667 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.655670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.656648 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.656650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.656739 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.656754 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 06:25:56.656830 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 06:25:56.656858 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":6,"thread_start_us":43,"threads_started":1}
I20260926 06:25:56.658351 12557 tablet_service.cc:1467] Processing CreateTablet for tablet 968cd662f8a341aa83492ab89a9b1075 (DEFAULT_TABLE table=test-table [id=75d42917668e4db3ab87bdf7ffdd1ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:56.658375 12691 tablet_service.cc:1467] Processing CreateTablet for tablet 968cd662f8a341aa83492ab89a9b1075 (DEFAULT_TABLE table=test-table [id=75d42917668e4db3ab87bdf7ffdd1ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:56.658375 12825 tablet_service.cc:1467] Processing CreateTablet for tablet 968cd662f8a341aa83492ab89a9b1075 (DEFAULT_TABLE table=test-table [id=75d42917668e4db3ab87bdf7ffdd1ffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:25:56.658631 12557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968cd662f8a341aa83492ab89a9b1075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:56.658702 12825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968cd662f8a341aa83492ab89a9b1075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:56.658702 12691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968cd662f8a341aa83492ab89a9b1075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:25:56.661216 12905 tablet_bootstrap.cc:492] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13: Bootstrap starting.
I20260926 06:25:56.661827 12906 tablet_bootstrap.cc:492] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: Bootstrap starting.
I20260926 06:25:56.661893 12905 tablet_bootstrap.cc:654] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:56.662189 12907 tablet_bootstrap.cc:492] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: Bootstrap starting.
I20260926 06:25:56.662463 12905 log.cc:824] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:56.662585 12906 tablet_bootstrap.cc:654] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:56.662817 12907 tablet_bootstrap.cc:654] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: Neither blocks nor log segments found. Creating new log.
I20260926 06:25:56.663106 12906 log.cc:824] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:56.663239 12905 tablet_bootstrap.cc:492] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13: No bootstrap required, opened a new log
I20260926 06:25:56.663317 12905 ts_tablet_manager.cc:1397] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:25:56.663354 12907 log.cc:824] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: Log is configured to *not* fsync() on all Append() calls
I20260926 06:25:56.663771 12906 tablet_bootstrap.cc:492] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: No bootstrap required, opened a new log
I20260926 06:25:56.663853 12906 ts_tablet_manager.cc:1397] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:25:56.664934 12905 raft_consensus.cc:348] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7454c1825184194a384344a62223a15" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34721 } } peers { permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 } } peers { permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 } }
I20260926 06:25:56.665066 12905 raft_consensus.cc:374] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:56.665087 12905 raft_consensus.cc:729] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a5068b39df14635be07560c05dc1a13, State: Initialized, Role: FOLLOWER
I20260926 06:25:56.665207 12905 consensus_queue.cc:260] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7454c1825184194a384344a62223a15" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34721 } } peers { permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 } } peers { permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 } }
I20260926 06:25:56.665522 12905 ts_tablet_manager.cc:1428] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:25:56.665529 12623 heartbeater.cc:503] Master 127.2.39.254:38169 was elected leader, sending a full tablet report...
I20260926 06:25:56.665606 12906 raft_consensus.cc:348] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7454c1825184194a384344a62223a15" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34721 } } peers { permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 } } peers { permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 } }
I20260926 06:25:56.665773 12906 raft_consensus.cc:374] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:56.665819 12906 raft_consensus.cc:729] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946fa748f3f04df08466693dfd0529d5, State: Initialized, Role: FOLLOWER
I20260926 06:25:56.665928 12906 consensus_queue.cc:260] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7454c1825184194a384344a62223a15" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34721 } } peers { permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 } } peers { permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 } }
I20260926 06:25:56.666218 12906 ts_tablet_manager.cc:1428] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:25:56.666277 12757 heartbeater.cc:503] Master 127.2.39.254:38169 was elected leader, sending a full tablet report...
I20260926 06:25:56.666291 12907 tablet_bootstrap.cc:492] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: No bootstrap required, opened a new log
I20260926 06:25:56.666342 12907 ts_tablet_manager.cc:1397] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 06:25:56.667870 12907 raft_consensus.cc:348] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7454c1825184194a384344a62223a15" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34721 } } peers { permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 } } peers { permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 } }
I20260926 06:25:56.667986 12907 raft_consensus.cc:374] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:25:56.668008 12907 raft_consensus.cc:729] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7454c1825184194a384344a62223a15, State: Initialized, Role: FOLLOWER
I20260926 06:25:56.668139 12907 consensus_queue.cc:260] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7454c1825184194a384344a62223a15" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34721 } } peers { permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 } } peers { permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 } }
I20260926 06:25:56.668429 12907 ts_tablet_manager.cc:1428] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:25:56.668442 12891 heartbeater.cc:503] Master 127.2.39.254:38169 was elected leader, sending a full tablet report...
W20260926 06:25:56.707334 12758 tablet.cc:2369] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:25:56.802088 12624 tablet.cc:2369] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:25:56.870707 12892 tablet.cc:2369] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:25:56.901068 12913 raft_consensus.cc:482] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:25:56.901187 12913 raft_consensus.cc:504] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7454c1825184194a384344a62223a15" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34721 } } peers { permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 } } peers { permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 } }
I20260926 06:25:56.901623 12913 leader_election.cc:288] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a5068b39df14635be07560c05dc1a13 (127.2.39.193:33995), 946fa748f3f04df08466693dfd0529d5 (127.2.39.194:33287)
I20260926 06:25:56.905582 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.902076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:59139 (local address 127.2.39.194:33287)
0926 06:25:56.902139 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.902141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.902210 (+ 69us) server_negotiation.cc:407] Connection header received
0926 06:25:56.902272 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.902273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.902300 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.902357 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:56.902539 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.903205 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.904657 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.905206 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.905283 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.905292 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.905452 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.905489 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:25:56.905556 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.901902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:60387 (local address 127.2.39.193:33995)
0926 06:25:56.901972 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.901974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.901990 (+ 16us) server_negotiation.cc:407] Connection header received
0926 06:25:56.902023 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.902024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.902046 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.902146 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:25:56.902327 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.903105 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.904658 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.905206 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.905283 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.905292 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.905452 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.905489 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 06:25:56.905594 12893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.901856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:33995 (local address 127.2.39.195:60387)
0926 06:25:56.901899 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:56.901912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:56.901961 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:56.902176 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:56.902177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:56.902179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:56.902251 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.902253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.903192 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.903193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.904564 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.904566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.905291 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.905292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.905371 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.905372 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:25:56.905431 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:25:56.905469 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":110}
I20260926 06:25:56.905607 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.901998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:33287 (local address 127.2.39.195:59139)
0926 06:25:56.902151 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:56.902161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:25:56.902211 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:25:56.902374 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:25:56.902374 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:25:56.902375 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:25:56.902456 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.902457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.903280 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.903280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.904564 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:25:56.904566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.905292 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.905292 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:25:56.905367 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.905371 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:25:56.905438 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:25:56.905469 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":32,"thread_start_us":107,"threads_started":1}
I20260926 06:25:56.905999 12577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968cd662f8a341aa83492ab89a9b1075" candidate_uuid: "c7454c1825184194a384344a62223a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a5068b39df14635be07560c05dc1a13" is_pre_election: true
I20260926 06:25:56.905999 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968cd662f8a341aa83492ab89a9b1075" candidate_uuid: "c7454c1825184194a384344a62223a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946fa748f3f04df08466693dfd0529d5" is_pre_election: true
I20260926 06:25:56.906160 12577 raft_consensus.cc:2454] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7454c1825184194a384344a62223a15 in term 0.
I20260926 06:25:56.906169 12711 raft_consensus.cc:2454] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7454c1825184194a384344a62223a15 in term 0.
I20260926 06:25:56.906394 12779 leader_election.cc:302] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a5068b39df14635be07560c05dc1a13, c7454c1825184194a384344a62223a15; no voters:
I20260926 06:25:56.906515 12913 raft_consensus.cc:2781] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:25:56.906611 12913 raft_consensus.cc:482] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:25:56.906648 12913 raft_consensus.cc:3037] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:56.907341 12913 raft_consensus.cc:504] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7454c1825184194a384344a62223a15" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34721 } } peers { permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 } } peers { permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 } }
I20260926 06:25:56.907480 12913 leader_election.cc:288] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [CANDIDATE]: Term 1 election: Requested vote from peers 4a5068b39df14635be07560c05dc1a13 (127.2.39.193:33995), 946fa748f3f04df08466693dfd0529d5 (127.2.39.194:33287)
I20260926 06:25:56.907683 12577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968cd662f8a341aa83492ab89a9b1075" candidate_uuid: "c7454c1825184194a384344a62223a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a5068b39df14635be07560c05dc1a13"
I20260926 06:25:56.907749 12577 raft_consensus.cc:3037] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:56.908373 12577 raft_consensus.cc:2454] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7454c1825184194a384344a62223a15 in term 1.
I20260926 06:25:56.908524 12779 leader_election.cc:302] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a5068b39df14635be07560c05dc1a13, c7454c1825184194a384344a62223a15; no voters:
I20260926 06:25:56.908463 12711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968cd662f8a341aa83492ab89a9b1075" candidate_uuid: "c7454c1825184194a384344a62223a15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "946fa748f3f04df08466693dfd0529d5"
I20260926 06:25:56.908603 12913 raft_consensus.cc:2781] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:25:56.908586 12711 raft_consensus.cc:3037] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:25:56.908757 12913 raft_consensus.cc:686] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 1 LEADER]: Becoming Leader. State: Replica: c7454c1825184194a384344a62223a15, State: Running, Role: LEADER
I20260926 06:25:56.908878 12913 consensus_queue.cc:237] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7454c1825184194a384344a62223a15" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34721 } } peers { permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 } } peers { permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 } }
I20260926 06:25:56.909433 12711 raft_consensus.cc:2454] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7454c1825184194a384344a62223a15 in term 1.
I20260926 06:25:56.909777 12432 catalog_manager.cc:5495] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 reported cstate change: term changed from 0 to 1, leader changed from <none> to c7454c1825184194a384344a62223a15 (127.2.39.195). New cstate: current_term: 1 leader_uuid: "c7454c1825184194a384344a62223a15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7454c1825184194a384344a62223a15" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 } health_report { overall_health: UNKNOWN } } }
I20260926 06:25:56.938372 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.935425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.2.39.195:34721)
0926 06:25:56.935510 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 06:25:56.935512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:25:56.935556 (+ 44us) server_negotiation.cc:407] Connection header received
0926 06:25:56.935665 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:25:56.935668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:25:56.935690 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:25:56.935769 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:25:56.935927 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.936405 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.937180 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:25:56.937369 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:25:56.937818 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:25:56.938000 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:25:56.938120 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:25:56.938218 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:25:56.938243 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 06:25:56.938263 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 06:25:56.940280 12711 raft_consensus.cc:1261] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5 [term 1 FOLLOWER]: Refusing update from remote peer c7454c1825184194a384344a62223a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:56.940297 12577 raft_consensus.cc:1261] T 968cd662f8a341aa83492ab89a9b1075 P 4a5068b39df14635be07560c05dc1a13 [term 1 FOLLOWER]: Refusing update from remote peer c7454c1825184194a384344a62223a15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:25:56.940631 12915 consensus_queue.cc:1035] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a5068b39df14635be07560c05dc1a13" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:56.940758 12913 consensus_queue.cc:1035] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "946fa748f3f04df08466693dfd0529d5" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 33287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:25:56.943629 12919 mvcc.cc:205] Tried to move back new op lower bound from 7333494607624294400 to 7333494607499214848. Current Snapshot: MvccSnapshot[applied={T|T < 7333494607624294400}]
I20260926 06:25:56.945065 12920 mvcc.cc:205] Tried to move back new op lower bound from 7333494607624294400 to 7333494607499214848. Current Snapshot: MvccSnapshot[applied={T|T < 7333494607624294400 or (T in {7333494607624294400})}]
I20260926 06:25:56.946362 12432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:56.946455 12432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:25:56.947230 12432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07e3fbf08581457ca085ad600c0ea16e: Sending DeleteTablet for 3 replicas of tablet 968cd662f8a341aa83492ab89a9b1075
I20260926 06:25:56.947554 12825 tablet_service.cc:1514] Processing DeleteTablet for tablet 968cd662f8a341aa83492ab89a9b1075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:56 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:40284
I20260926 06:25:56.947582 12557 tablet_service.cc:1514] Processing DeleteTablet for tablet 968cd662f8a341aa83492ab89a9b1075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:56 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:49548
I20260926 06:25:56.947631 12691 tablet_service.cc:1514] Processing DeleteTablet for tablet 968cd662f8a341aa83492ab89a9b1075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:25:56 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:38012
I20260926 06:25:56.947892 12929 tablet_replica.cc:330] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: stopping tablet replica
I20260926 06:25:56.947993 12929 raft_consensus.cc:2229] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:25:56.947991 12927 tablet_replica.cc:330] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: stopping tablet replica
I20260926 06:25:56.948107 12929 raft_consensus.cc:2258] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:56.948134 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 12494
I20260926 06:25:56.948863 12929 ts_tablet_manager.cc:1905] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:56.948854 12927 raft_consensus.cc:2229] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:25:56.949054 12927 raft_consensus.cc:2258] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:25:56.949523 12927 ts_tablet_manager.cc:1905] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:25:56.950769 12929 ts_tablet_manager.cc:1918] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:56.950845 12929 log.cc:1196] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/968cd662f8a341aa83492ab89a9b1075
I20260926 06:25:56.951114 12929 ts_tablet_manager.cc:1939] T 968cd662f8a341aa83492ab89a9b1075 P 946fa748f3f04df08466693dfd0529d5: Deleting consensus metadata
I20260926 06:25:56.951499 12417 catalog_manager.cc:4841] TS 946fa748f3f04df08466693dfd0529d5 (127.2.39.194:33287): tablet 968cd662f8a341aa83492ab89a9b1075 (table test-table [id=75d42917668e4db3ab87bdf7ffdd1ffb]) successfully deleted
I20260926 06:25:56.951829 12927 ts_tablet_manager.cc:1918] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:25:56.951900 12927 log.cc:1196] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/968cd662f8a341aa83492ab89a9b1075
I20260926 06:25:56.952176 12927 ts_tablet_manager.cc:1939] T 968cd662f8a341aa83492ab89a9b1075 P c7454c1825184194a384344a62223a15: Deleting consensus metadata
I20260926 06:25:56.952751 12418 catalog_manager.cc:4841] TS c7454c1825184194a384344a62223a15 (127.2.39.195:34721): tablet 968cd662f8a341aa83492ab89a9b1075 (table test-table [id=75d42917668e4db3ab87bdf7ffdd1ffb]) successfully deleted
W20260926 06:25:56.955463 12417 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:33995 (error 108)
I20260926 06:25:56.955754 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 12628
I20260926 06:25:56.956351 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:25:56.955930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:33995 (local address 127.0.0.1:49556)
0926 06:25:56.956264 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 06:25:56.956311 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:33995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311}
W20260926 06:25:56.957206 12417 catalog_manager.cc:4584] TS 4a5068b39df14635be07560c05dc1a13 (127.2.39.193:33995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 968cd662f8a341aa83492ab89a9b1075: Network error: Client connection negotiation failed: client connection to 127.2.39.193:33995: connect: Connection refused (error 111)
I20260926 06:25:56.962021 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 12762
I20260926 06:25:56.966506 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 12400
2026-09-26T06:25:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2144 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:25:57 dist-test-slave-fkbn krb5kdc[12933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:25:57 dist-test-slave-fkbn krb5kdc[12933](info): set up 2 sockets
Sep 26 06:25:57 dist-test-slave-fkbn krb5kdc[12933](info): commencing operation
krb5kdc: starting...
W20260926 06:25:58.010526 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:25:58 dist-test-slave-fkbn krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:25:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:25:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.39.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.39.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/rangeradmin_127.2.39.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/rangeradmin_127.2.39.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.39.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.39.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/rangerlookup_127.2.39.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/rangerlookup_127.2.39.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.39.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.39.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/HTTP_127.2.39.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/HTTP_127.2.39.212@KRBTEST.COM.keytab.
I20260926 06:25:58.172725 2207 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOgNaKF/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/postgres -l logfile start
2026-09-26 06:26:00.764 UTC [12963] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 06:26:00.764 UTC [12963] LOG: listening on IPv4 address "127.2.39.212", port 47267
2026-09-26 06:26:00.767 UTC [12963] LOG: listening on Unix socket "/tmp/.s.PGSQL.47267"
2026-09-26 06:26:00.789 UTC [12966] LOG: database system was shut down at 2026-09-26 06:25:59 UTC
2026-09-26 06:26:00.793 UTC [12963] LOG: database system is ready to accept connections
I20260926 06:26:01.746611 2207 mini_postgres.cc:96] Postgres bound to 47267
2026-09-26 06:26:01.752 UTC [12974] FATAL: database "slave" does not exist
127.2.39.212:47267 - accepting connections
I20260926 06:26:01.753327 2207 mini_ranger.cc:161] Starting Ranger...
I20260926 06:26:01.767724 2207 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 06:26:02.270918 2207 mini_ranger.cc:87] Created ranger Postgres database
I20260926 06:26:02.271023 2207 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-admin
2026-09-26 06:26:02,540 [I] DB FLAVOR :POSTGRES
2026-09-26 06:26:02,541 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:26:02,541 [I] Checking connection..
2026-09-26 06:26:02,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 06:26:02,771 [I] Checking connection passed.
2026-09-26 06:26:02,772 [I] --------- Verifying version history table ---------
2026-09-26 06:26:02,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:26:03,039 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:26:03,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:26:03,294 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:26:03,295 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 06:26:03,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 06:26:03.530 UTC [13038] WARNING: there is no transaction in progress
2026-09-26 06:26:03,536 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 06:26:03,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:26:03,765 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 06:26:03,766 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:26:03,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 06:26:04,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 06:26:04,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fkbn.c.gcp-upstream.internal','N') ;"
2026-09-26 06:26:04,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:26:04,748 [I] Table x_portal_user does not exist in database ranger
2026-09-26 06:26:04,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:26:04,983 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 06:26:04,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:26:05,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:26:05,456 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 06:26:05,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 06:26:06.431 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.447 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.462 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.478 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.493 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.513 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.567 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.573 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.587 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.598 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.608 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.621 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.630 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.637 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.643 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:06.650 UTC [13146] WARNING: there is no transaction in progress
2026-09-26 06:26:07,303 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 06:26:07,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 06:26:07,550 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 06:26:07,550 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:26:07,800 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 06:26:07,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:26:08,054 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 06:26:08,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:26:08,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:26:08,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fkbn.c.gcp-upstream.internal';"
2026-09-26 06:26:08,771 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 06:26:08,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 06:26:09,002 [I] DB_PATCHES have already been applied
I20260926 06:26:09.008718 2207 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOgNaKF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 06:26:09.008786 2207 mini_ranger.cc:193] Using host: 127.2.39.212
I20260926 06:26:09.011852 2207 mini_ranger.cc:227] Ranger admin URL: http://127.2.39.212:55729
Sep 26, 2026 6:26:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:26:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:26:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 6:26:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 6:26:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 6:26:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 6:26:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 06:26:10.122093 2207 mini_ranger.cc:160] Time spent starting Ranger: real 8.369s user 0.004s sys 0.005s
Sep 26, 2026 6:26:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 6:26:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 26, 2026 6:26:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 6:26:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 6:26:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 06:26:30.630203 2207 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.39.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.39.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/rangerkms_127.2.39.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/rangerkms_127.2.39.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.39.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.39.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.39.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/HTTP_127.2.39.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.39.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/HTTP_127.2.39.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 06:26:30 dist-test-slave-fkbn krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403990, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 06:26:30.708490 2207 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 06:26:30.717283 2207 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 06:26:31.624560 2207 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 06:26:31.624642 2207 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-kms
I20260926 06:26:31.624660 2207 mini_ranger_kms.cc:227] Using postgres at 127.2.39.212:47267
2026-09-26 06:26:31,836 [I] DB FLAVOR :POSTGRES
2026-09-26 06:26:31,836 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:26:31,836 [I] Checking connection
2026-09-26 06:26:31,836 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 06:26:32,098 [I] connection success
2026-09-26 06:26:32,098 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 06:26:32,098 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 06:26:32,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 06:26:32,348 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 06:26:32,348 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:26:32,348 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 06:26:32,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:47267/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 06:26:32,653 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 06:26:33.118181 2207 mini_ranger_kms.cc:310] Created kms service
I20260926 06:26:33.329898 2207 mini_ranger_kms.cc:326] Created kudu user
I20260926 06:26:33.373000 2207 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 06:26:33 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403990, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:26:33.768849 2207 mini_ranger_kms.cc:384] Added ranger policy
I20260926 06:26:33.768968 2207 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 06:26:33.768989 2207 mini_ranger_kms.cc:240] Using host: 127.2.39.212
I20260926 06:26:33.772251 2207 mini_ranger_kms.cc:276] Ranger KMS PID: 13326
I20260926 06:26:33.772377 2207 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.39.212:55765
Sep 26, 2026 6:26:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:26:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 6:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 6:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 6:26:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 06:26:36.041508 2207 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.333s user 0.017s sys 0.176s
I20260926 06:26:36.041626 2207 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 06:26:36.041704 2207 mini_ranger_kms.cc:401] 127.2.39.212:55765/kms/v1/keys
Sep 26 06:26:36 dist-test-slave-fkbn krb5kdc[12933](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790403996, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.39.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 06:26:36 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403996, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.39.212@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
Sep 26 06:26:36 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403996, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.39.212@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
Sep 26 06:26:39 dist-test-slave-fkbn krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:26:39.110162 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38417
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:38417
--ranger_config_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:55765/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:26:39.211241 13377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:26:39.211468 13377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:26:39.211493 13377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:26:39.212678 13377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:26:39.212730 13377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:26:39.212749 13377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:26:39.212762 13377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:26:39.212776 13377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:26:39.214460 13377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.39.212:55765/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:38417
--ranger_config_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:26:39.214715 13377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:26:39.214988 13377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:26:39.221159 13382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:26:39.221253 13383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:26:39.221319 13385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:26:39.221223 13377 server_base.cc:1034] running on GCE node
I20260926 06:26:39.221668 13377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:26:39.221979 13377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:26:39.223148 13377 hybrid_clock.cc:648] HybridClock initialized: now 1790403999223136 us; error 30 us; skew 500 ppm
Sep 26 06:26:39 dist-test-slave-fkbn krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403999, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:26:39.225868 13377 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:26:39.226298 13377 webserver.cc:460] Webserver started at http://127.2.39.254:44145/ using document root <none> and password file <none>
I20260926 06:26:39.226521 13377 fs_manager.cc:359] Metadata directory not provided
I20260926 06:26:39.226586 13377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:26:39.226747 13377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:26:39 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403999, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:26:39.271281 13377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "0978ff28732d4f7f8b073b05e2b78b40"
format_stamp: "Formatted at 2026-09-26 06:26:39 on dist-test-slave-fkbn"
server_key: "b582dc17938298f468ba41fa901fa67d"
server_key_iv: "1da0eaf308452032981cd0177d801859"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:39.271713 13377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "0978ff28732d4f7f8b073b05e2b78b40"
format_stamp: "Formatted at 2026-09-26 06:26:39 on dist-test-slave-fkbn"
server_key: "b582dc17938298f468ba41fa901fa67d"
server_key_iv: "1da0eaf308452032981cd0177d801859"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:39.273152 13377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 06:26:39.284531 13392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:26:39.284775 13377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:26:39.284894 13377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "0978ff28732d4f7f8b073b05e2b78b40"
format_stamp: "Formatted at 2026-09-26 06:26:39 on dist-test-slave-fkbn"
server_key: "b582dc17938298f468ba41fa901fa67d"
server_key_iv: "1da0eaf308452032981cd0177d801859"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:39.284989 13377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:26:39.298779 13377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:26:39.300853 13377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:26:39.301043 13377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:26:39.306564 13377 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:38417
I20260926 06:26:39.306607 13444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:38417 every 8 connection(s)
I20260926 06:26:39.307014 13377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:26:39.307730 13445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:26:39.310022 13445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40: Bootstrap starting.
I20260926 06:26:39.310595 13445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40: Neither blocks nor log segments found. Creating new log.
I20260926 06:26:39.310878 13445 log.cc:824] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40: Log is configured to *not* fsync() on all Append() calls
I20260926 06:26:39.311544 13445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40: No bootstrap required, opened a new log
I20260926 06:26:39.313494 13445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0978ff28732d4f7f8b073b05e2b78b40" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38417 } }
I20260926 06:26:39.313727 13445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:26:39.313778 13445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0978ff28732d4f7f8b073b05e2b78b40, State: Initialized, Role: FOLLOWER
I20260926 06:26:39.313936 13445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0978ff28732d4f7f8b073b05e2b78b40" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38417 } }
I20260926 06:26:39.314033 13445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:26:39.314076 13445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:26:39.314123 13445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:26:39.314628 13445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0978ff28732d4f7f8b073b05e2b78b40" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38417 } }
I20260926 06:26:39.314739 13445 leader_election.cc:302] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0978ff28732d4f7f8b073b05e2b78b40; no voters:
I20260926 06:26:39.314913 13445 leader_election.cc:288] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:26:39.314956 13448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:26:39.315164 13445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:26:39.315182 13448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [term 1 LEADER]: Becoming Leader. State: Replica: 0978ff28732d4f7f8b073b05e2b78b40, State: Running, Role: LEADER
I20260926 06:26:39.315445 13448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0978ff28732d4f7f8b073b05e2b78b40" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38417 } }
I20260926 06:26:39.315815 13448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0978ff28732d4f7f8b073b05e2b78b40. Latest consensus state: current_term: 1 leader_uuid: "0978ff28732d4f7f8b073b05e2b78b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0978ff28732d4f7f8b073b05e2b78b40" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38417 } } }
I20260926 06:26:39.315826 13450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0978ff28732d4f7f8b073b05e2b78b40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0978ff28732d4f7f8b073b05e2b78b40" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 38417 } } }
I20260926 06:26:39.315991 13450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [sys.catalog]: This master's current role is: LEADER
I20260926 06:26:39.315976 13448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40 [sys.catalog]: This master's current role is: LEADER
I20260926 06:26:39.316215 13457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:26:39.316512 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 13377
I20260926 06:26:39.316627 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:26:39.316807 2207 external_mini_cluster.cc:1428] Setting key 9fa8f63db9a8b2de42906bd0ba358c57
I20260926 06:26:39.317447 13457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:26:39.317852 13445 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 26 06:26:39 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790403999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:26:39.327593 13457 catalog_manager.cc:1347] Generated new cluster ID: b116c0a7936a44519a2b7b488710b3a0
I20260926 06:26:39.327694 13457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:26:39.332551 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:39.317236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.2.39.254:38417)
0926 06:26:39.318134 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:39.318144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:39.319407 (+ 1263us) server_negotiation.cc:407] Connection header received
0926 06:26:39.319964 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:39.319973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:39.320095 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:39.320228 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:26:39.321302 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:39.323050 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:39.324125 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:39.324910 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:39.325047 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:39.328177 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:26:39.328181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:26:39.328181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:26:39.328193 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:26:39.330524 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:26:39.330929 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:26:39.330932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:26:39.330933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:26:39.331029 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:26:39.331221 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:26:39.331222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:26:39.331222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:26:39.331707 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:26:39.331848 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:39.332112 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 06:26:39.332259 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":66,"threads_started":1}
I20260926 06:26:39.347090 13457 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:26:39.347836 13457 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:26:39.350198 13457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40: Generated new TSK 0
I20260926 06:26:39.350416 13457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:26:40.052981 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:55765/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38417
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:26:40.166225 13490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:26:40.166508 13490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:26:40.166544 13490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:26:40.168220 13490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:26:40.168274 13490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:26:40.168313 13490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:26:40.170433 13490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.39.212:55765/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:26:40.170677 13490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:26:40.170961 13490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:26:40.176573 13496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:26:40.176559 13495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:26:40.176561 13498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:26:40.176880 13490 server_base.cc:1034] running on GCE node
I20260926 06:26:40.177086 13490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:26:40.177372 13490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:26:40.178546 13490 hybrid_clock.cc:648] HybridClock initialized: now 1790404000178527 us; error 34 us; skew 500 ppm
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:26:40.181396 13490 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:26:40.181892 13490 webserver.cc:460] Webserver started at http://127.2.39.193:43899/ using document root <none> and password file <none>
I20260926 06:26:40.182137 13490 fs_manager.cc:359] Metadata directory not provided
I20260926 06:26:40.182206 13490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:26:40.182351 13490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:26:40.196533 13490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "1e0cae01b02541cc903c8c8106e557cb"
format_stamp: "Formatted at 2026-09-26 06:26:40 on dist-test-slave-fkbn"
server_key: "044aa67f92d7cf9c0ea395f159f59e78"
server_key_iv: "011ccf46fe670a516fb2caf3fb084fb7"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:40.197734 13490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "1e0cae01b02541cc903c8c8106e557cb"
format_stamp: "Formatted at 2026-09-26 06:26:40 on dist-test-slave-fkbn"
server_key: "044aa67f92d7cf9c0ea395f159f59e78"
server_key_iv: "011ccf46fe670a516fb2caf3fb084fb7"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:40.199347 13490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:26:40.216801 13507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:26:40.217078 13490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:26:40.217214 13490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "1e0cae01b02541cc903c8c8106e557cb"
format_stamp: "Formatted at 2026-09-26 06:26:40 on dist-test-slave-fkbn"
server_key: "044aa67f92d7cf9c0ea395f159f59e78"
server_key_iv: "011ccf46fe670a516fb2caf3fb084fb7"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:40.217355 13490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:26:40.239531 13490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:26:40.242228 13490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:26:40.242486 13490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:26:40.242802 13490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:26:40.243235 13490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:26:40.243352 13490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:26:40.243453 13490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:26:40.243532 13490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:26:40.251205 13490 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:38881
I20260926 06:26:40.251718 13490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:26:40.255965 13620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:38881 every 8 connection(s)
I20260926 06:26:40.259507 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 13490
I20260926 06:26:40.259613 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:26:40.259820 2207 external_mini_cluster.cc:1428] Setting key 2e608c55b8fde5b62489bfdb73dfb452
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:26:40.268788 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.257378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38417 (local address 127.2.39.193:38657)
0926 06:26:40.257671 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 06:26:40.257718 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:26:40.258383 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:26:40.258823 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:26:40.258827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:26:40.259087 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:26:40.259376 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.259382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.260456 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.260457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.261820 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.261823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.262426 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.262427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.262580 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.263190 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:26:40.263200 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:26:40.264855 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:26:40.267046 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:26:40.267052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:26:40.267055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:26:40.267372 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:26:40.267624 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:26:40.267625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:26:40.267625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:26:40.267760 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:26:40.268110 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:26:40.268117 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:26:40.268272 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 06:26:40.268397 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 06:26:40.268551 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":78,"threads_started":1}
I20260926 06:26:40.269124 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.257382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:38657 (local address 127.2.39.254:38417)
0926 06:26:40.258584 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:40.258587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:40.258595 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:26:40.258621 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:40.258622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:40.258633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:40.258710 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:26:40.259512 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.260316 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.261933 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.262254 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.262343 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.264969 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:26:40.264971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:26:40.264971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:26:40.264979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:26:40.266945 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:26:40.267471 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:26:40.267473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:26:40.267473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:26:40.267545 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:26:40.267862 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:26:40.267863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:26:40.267863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:26:40.268009 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:26:40.268080 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:40.268934 (+ 854us) server_negotiation.cc:299] Negotiation successful
0926 06:26:40.269021 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1171,"thread_start_us":83,"threads_started":1}
I20260926 06:26:40.269629 13621 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38417
I20260926 06:26:40.269732 13621 heartbeater.cc:464] Registering TS with master...
I20260926 06:26:40.269917 13621 heartbeater.cc:511] Master 127.2.39.254:38417 requested a full tablet report, sending...
I20260926 06:26:40.270465 13409 ts_manager.cc:194] Registered new tserver with Master: 1e0cae01b02541cc903c8c8106e557cb (127.2.39.193:38881)
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
I20260926 06:26:40.271325 13409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:38657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:26:40.305016 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:55765/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38417
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 06:26:40.437214 13627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:26:40.437552 13627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:26:40.437652 13627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:26:40.439452 13627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:26:40.439565 13627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:26:40.439643 13627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:26:40.441927 13627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.39.212:55765/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:26:40.442369 13627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:26:40.442674 13627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:26:40.449486 13635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:26:40.450340 13634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:26:40.450655 13627 server_base.cc:1034] running on GCE node
W20260926 06:26:40.454020 13637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:26:40.454300 13627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:26:40.454550 13627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:26:40.455696 13627 hybrid_clock.cc:648] HybridClock initialized: now 1790404000455663 us; error 47 us; skew 500 ppm
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:26:40.459229 13627 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:26:40.459718 13627 webserver.cc:460] Webserver started at http://127.2.39.194:39891/ using document root <none> and password file <none>
I20260926 06:26:40.459972 13627 fs_manager.cc:359] Metadata directory not provided
I20260926 06:26:40.460021 13627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:26:40.460143 13627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:26:40.483912 13627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "d8ebf44ca3a2472fb83e7d185e5953ce"
format_stamp: "Formatted at 2026-09-26 06:26:40 on dist-test-slave-fkbn"
server_key: "9de6051df917afd363e5bc94e1b309df"
server_key_iv: "4cd8265dc422b4d145b2f4d0d7c25e4f"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:40.484484 13627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "d8ebf44ca3a2472fb83e7d185e5953ce"
format_stamp: "Formatted at 2026-09-26 06:26:40 on dist-test-slave-fkbn"
server_key: "9de6051df917afd363e5bc94e1b309df"
server_key_iv: "4cd8265dc422b4d145b2f4d0d7c25e4f"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:40.489506 13627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.000s
I20260926 06:26:40.510320 13645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:26:40.510617 13627 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260926 06:26:40.510752 13627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "d8ebf44ca3a2472fb83e7d185e5953ce"
format_stamp: "Formatted at 2026-09-26 06:26:40 on dist-test-slave-fkbn"
server_key: "9de6051df917afd363e5bc94e1b309df"
server_key_iv: "4cd8265dc422b4d145b2f4d0d7c25e4f"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:40.510924 13627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:26:40.532002 13627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:26:40.534322 13627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:26:40.534457 13627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:26:40.534683 13627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:26:40.535061 13627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:26:40.535149 13627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:26:40.535205 13627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:26:40.535247 13627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:26:40.542434 13627 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:45289
I20260926 06:26:40.542913 13627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:26:40.544938 13758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:45289 every 8 connection(s)
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:26:40.551996 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 13627
I20260926 06:26:40.552112 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:26:40.552325 2207 external_mini_cluster.cc:1428] Setting key b7cc2f37d33d85f949cf96becb9923f5
I20260926 06:26:40.555495 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.545530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:57511 (local address 127.2.39.254:38417)
0926 06:26:40.545593 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:40.545594 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:40.545605 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:26:40.546212 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:40.546213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:40.546231 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:40.546260 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:26:40.547099 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.547863 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.549318 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.549707 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.549789 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.552272 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:26:40.552274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:26:40.552275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:26:40.552282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:26:40.553995 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:26:40.554451 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:26:40.554452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:26:40.554452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:26:40.554510 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:26:40.554765 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:26:40.554766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:26:40.554767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:26:40.554897 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:26:40.554948 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:40.555322 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 06:26:40.555405 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:26:40.556216 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.545227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38417 (local address 127.2.39.194:57511)
0926 06:26:40.545427 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 06:26:40.545471 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:26:40.546081 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:26:40.546459 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:26:40.546463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:26:40.546694 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:26:40.546974 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.546980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.547990 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.547991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.549209 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.549212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.549820 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.549822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.549974 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.550574 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:26:40.550584 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:26:40.552162 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:26:40.554087 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:26:40.554094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:26:40.554096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:26:40.554364 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:26:40.554579 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:26:40.554580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:26:40.554580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:26:40.554699 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:26:40.554991 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:26:40.554996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:26:40.555178 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 06:26:40.555806 (+ 628us) client_negotiation.cc:240] Negotiation successful
0926 06:26:40.555995 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260926 06:26:40.556818 13759 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38417
I20260926 06:26:40.556912 13759 heartbeater.cc:464] Registering TS with master...
I20260926 06:26:40.557080 13759 heartbeater.cc:511] Master 127.2.39.254:38417 requested a full tablet report, sending...
I20260926 06:26:40.557442 13409 ts_manager.cc:194] Registered new tserver with Master: d8ebf44ca3a2472fb83e7d185e5953ce (127.2.39.194:45289)
I20260926 06:26:40.557893 13409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:57511
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:26:40.601650 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:55765/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38417
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:26:40.727159 13764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:26:40.727381 13764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:26:40.727420 13764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:26:40.728586 13764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:26:40.728655 13764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:26:40.728699 13764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:26:40.730252 13764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.39.212:55765/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:26:40.730512 13764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:26:40.730778 13764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:26:40.744779 13769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:26:40.744787 13770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:26:40.745187 13772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:26:40.747838 13764 server_base.cc:1034] running on GCE node
I20260926 06:26:40.748123 13764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:26:40.748481 13764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:26:40.749701 13764 hybrid_clock.cc:648] HybridClock initialized: now 1790404000749676 us; error 37 us; skew 500 ppm
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:26:40.753014 13764 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:26:40.753563 13764 webserver.cc:460] Webserver started at http://127.2.39.195:36177/ using document root <none> and password file <none>
I20260926 06:26:40.753984 13764 fs_manager.cc:359] Metadata directory not provided
I20260926 06:26:40.754099 13764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:26:40.754279 13764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:26:40.768523 13764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "525be0a98f144ed2a352c7e5b15266d0"
format_stamp: "Formatted at 2026-09-26 06:26:40 on dist-test-slave-fkbn"
server_key: "5eafb855f84c224ac0b0aedd3c91c637"
server_key_iv: "28c6a2a11525fccfe6a05b0ce0ac01f2"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:40.769066 13764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "525be0a98f144ed2a352c7e5b15266d0"
format_stamp: "Formatted at 2026-09-26 06:26:40 on dist-test-slave-fkbn"
server_key: "5eafb855f84c224ac0b0aedd3c91c637"
server_key_iv: "28c6a2a11525fccfe6a05b0ce0ac01f2"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:40.773989 13764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.001s
I20260926 06:26:40.795920 13779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:26:40.796227 13764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 06:26:40.796327 13764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "525be0a98f144ed2a352c7e5b15266d0"
format_stamp: "Formatted at 2026-09-26 06:26:40 on dist-test-slave-fkbn"
server_key: "5eafb855f84c224ac0b0aedd3c91c637"
server_key_iv: "28c6a2a11525fccfe6a05b0ce0ac01f2"
server_key_version: "kuduclusterkey@0"
I20260926 06:26:40.796433 13764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:26:40.820220 13764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:26:40.822896 13764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:26:40.823107 13764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:26:40.823427 13764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:26:40.823884 13764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:26:40.823983 13764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:26:40.824062 13764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:26:40.824124 13764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:26:40.832223 13764 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:38797
I20260926 06:26:40.832736 13764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:26:40.838574 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 13764
I20260926 06:26:40.838694 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:26:40.838909 2207 external_mini_cluster.cc:1428] Setting key 7485927fd2660860ea9a84f716bbec1d
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:26:40.847934 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.836549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:37859 (local address 127.2.39.254:38417)
0926 06:26:40.836601 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:40.836604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:40.836991 (+ 387us) server_negotiation.cc:407] Connection header received
0926 06:26:40.837774 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:40.837775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:40.837790 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:40.837828 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:26:40.838884 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.839601 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.841154 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.841435 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.841488 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.844361 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:26:40.844362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:26:40.844363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:26:40.844370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:26:40.846240 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:26:40.846744 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:26:40.846745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:26:40.846745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:26:40.846804 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:26:40.847116 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:26:40.847117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:26:40.847117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:26:40.847255 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:26:40.847305 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:40.847714 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 06:26:40.847805 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:26:40.835146 13892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:38797 every 8 connection(s)
I20260926 06:26:40.848819 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.836241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:38417 (local address 127.2.39.195:37859)
0926 06:26:40.836846 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0926 06:26:40.836895 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:26:40.837623 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:26:40.838047 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:26:40.838052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:26:40.838318 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:26:40.838619 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.838625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.839791 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.839792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.841037 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.841040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.841627 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.841628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.841774 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.842428 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:26:40.842439 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:26:40.844235 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:26:40.846347 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:26:40.846354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:26:40.846356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:26:40.846647 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:26:40.846898 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:26:40.846898 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:26:40.846899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:26:40.847023 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:26:40.847363 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:26:40.847370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:26:40.847531 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 06:26:40.848398 (+ 867us) client_negotiation.cc:240] Negotiation successful
0926 06:26:40.848575 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":91,"threads_started":1}
I20260926 06:26:40.850642 13893 heartbeater.cc:347] Connected to a master server at 127.2.39.254:38417
I20260926 06:26:40.850739 13893 heartbeater.cc:464] Registering TS with master...
I20260926 06:26:40.850936 13893 heartbeater.cc:511] Master 127.2.39.254:38417 requested a full tablet report, sending...
I20260926 06:26:40.851406 13409 ts_manager.cc:194] Registered new tserver with Master: 525be0a98f144ed2a352c7e5b15266d0 (127.2.39.195:38797)
I20260926 06:26:40.851814 13409 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:37859
I20260926 06:26:40.861495 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:26:40.871477 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.863116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.2.39.254:38417)
0926 06:26:40.863176 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:40.863178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:40.864172 (+ 994us) server_negotiation.cc:407] Connection header received
0926 06:26:40.864289 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:40.864290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:40.864304 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:40.864353 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:26:40.865196 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.865994 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.866850 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.867063 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.867146 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.868641 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:26:40.868643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:26:40.868643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:26:40.868650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:26:40.870162 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:26:40.870558 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:26:40.870559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:26:40.870560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:26:40.870624 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:26:40.870838 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:26:40.870839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:26:40.870839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:26:40.870982 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:26:40.871086 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:40.871308 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 06:26:40.871386 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 06:26:40.873778 13409 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:26:40.875507 13409 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:26:40.885242 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.880026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:38797 (local address 127.0.0.1:50642)
0926 06:26:40.880789 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 06:26:40.880796 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 06:26:40.880861 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:26:40.881143 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:26:40.881147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:26:40.881149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:26:40.881217 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.881218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.882562 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.882563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.883848 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.883851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.884933 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.884934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.885019 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.885033 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 06:26:40.885107 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:26:40.885158 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":63,"threads_started":1}
I20260926 06:26:40.885774 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.880274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.2.39.195:38797)
0926 06:26:40.880437 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:40.880442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:40.880866 (+ 424us) server_negotiation.cc:407] Connection header received
0926 06:26:40.880941 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:40.880944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:40.880962 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:40.881042 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:26:40.881305 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.882473 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.883983 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.884727 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.884806 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.884894 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:40.885635 (+ 741us) server_negotiation.cc:299] Negotiation successful
0926 06:26:40.885682 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260926 06:26:40.886631 13826 tablet_service.cc:1467] Processing CreateTablet for tablet 0723323a84fc4e5f8a26ce2bc302c624 (DEFAULT_TABLE table=test-table [id=8b2a361fa7cc45fe84f090e218b7e5f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:26:40.887015 13826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0723323a84fc4e5f8a26ce2bc302c624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:26:40.887820 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.879565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:45289 (local address 127.0.0.1:42182)
0926 06:26:40.880150 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 06:26:40.880195 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 06:26:40.880264 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:26:40.881208 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:26:40.881208 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:26:40.881210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:26:40.881241 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.881242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.882865 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.882866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.886591 (+ 3725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.886594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.887598 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.887599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.887665 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.887670 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:26:40.887706 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 06:26:40.887744 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":82,"threads_started":1}
I20260926 06:26:40.888243 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.879675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.2.39.194:45289)
0926 06:26:40.880991 (+ 1316us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:40.880996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:40.881005 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:26:40.881044 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:40.881046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:40.881061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:40.881131 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:26:40.881320 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.882744 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.886733 (+ 3989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.887486 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.887999 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.888089 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:40.888117 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 06:26:40.888158 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1278,"thread_start_us":107,"threads_started":1}
I20260926 06:26:40.888908 13674 tablet_service.cc:1467] Processing CreateTablet for tablet 0723323a84fc4e5f8a26ce2bc302c624 (DEFAULT_TABLE table=test-table [id=8b2a361fa7cc45fe84f090e218b7e5f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:26:40.889267 13674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0723323a84fc4e5f8a26ce2bc302c624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:26:40.892735 13912 tablet_bootstrap.cc:492] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: Bootstrap starting.
I20260926 06:26:40.893875 13912 tablet_bootstrap.cc:654] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: Neither blocks nor log segments found. Creating new log.
I20260926 06:26:40.894173 13912 log.cc:824] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: Log is configured to *not* fsync() on all Append() calls
I20260926 06:26:40.894892 13912 tablet_bootstrap.cc:492] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: No bootstrap required, opened a new log
I20260926 06:26:40.894975 13912 ts_tablet_manager.cc:1397] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:26:40.896430 13912 raft_consensus.cc:348] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 } } peers { permanent_uuid: "525be0a98f144ed2a352c7e5b15266d0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38797 } } peers { permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 } }
I20260926 06:26:40.896636 13912 raft_consensus.cc:374] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:26:40.896706 13912 raft_consensus.cc:729] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8ebf44ca3a2472fb83e7d185e5953ce, State: Initialized, Role: FOLLOWER
I20260926 06:26:40.896883 13912 consensus_queue.cc:260] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 } } peers { permanent_uuid: "525be0a98f144ed2a352c7e5b15266d0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38797 } } peers { permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 } }
I20260926 06:26:40.897171 13912 ts_tablet_manager.cc:1428] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:26:40.897189 13759 heartbeater.cc:503] Master 127.2.39.254:38417 was elected leader, sending a full tablet report...
I20260926 06:26:40.904942 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.879927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.2.39.193:38881)
0926 06:26:40.901672 (+ 21745us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:40.901678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:40.901692 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:26:40.901727 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:40.901730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:40.901747 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:40.901855 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:26:40.902019 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.902694 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.904054 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.904575 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.904678 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.904735 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:40.904826 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 06:26:40.904869 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21705,"thread_start_us":55,"threads_started":1}
I20260926 06:26:40.904934 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.879817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:38881 (local address 127.0.0.1:37532)
0926 06:26:40.880577 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0926 06:26:40.880587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:26:40.880683 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:26:40.901865 (+ 21182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:26:40.901866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:26:40.901870 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:26:40.901922 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.901924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.902807 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.902809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.903946 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.903949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.904682 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.904683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.904745 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.904751 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:26:40.904813 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:26:40.904858 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":74,"threads_started":1}
I20260926 06:26:40.905745 13911 tablet_bootstrap.cc:492] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: Bootstrap starting.
I20260926 06:26:40.905689 13552 tablet_service.cc:1467] Processing CreateTablet for tablet 0723323a84fc4e5f8a26ce2bc302c624 (DEFAULT_TABLE table=test-table [id=8b2a361fa7cc45fe84f090e218b7e5f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:26:40.906034 13552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0723323a84fc4e5f8a26ce2bc302c624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:26:40.906744 13911 tablet_bootstrap.cc:654] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: Neither blocks nor log segments found. Creating new log.
I20260926 06:26:40.907981 13911 log.cc:824] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: Log is configured to *not* fsync() on all Append() calls
I20260926 06:26:40.910482 13916 tablet_bootstrap.cc:492] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb: Bootstrap starting.
I20260926 06:26:40.911217 13916 tablet_bootstrap.cc:654] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb: Neither blocks nor log segments found. Creating new log.
I20260926 06:26:40.911566 13916 log.cc:824] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb: Log is configured to *not* fsync() on all Append() calls
I20260926 06:26:40.912431 13911 tablet_bootstrap.cc:492] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: No bootstrap required, opened a new log
I20260926 06:26:40.912562 13911 ts_tablet_manager.cc:1397] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 06:26:40.914186 13916 tablet_bootstrap.cc:492] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb: No bootstrap required, opened a new log
I20260926 06:26:40.914297 13916 ts_tablet_manager.cc:1397] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 06:26:40.914923 13911 raft_consensus.cc:348] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 } } peers { permanent_uuid: "525be0a98f144ed2a352c7e5b15266d0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38797 } } peers { permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 } }
I20260926 06:26:40.915237 13911 raft_consensus.cc:374] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:26:40.915349 13911 raft_consensus.cc:729] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525be0a98f144ed2a352c7e5b15266d0, State: Initialized, Role: FOLLOWER
I20260926 06:26:40.915544 13911 consensus_queue.cc:260] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 } } peers { permanent_uuid: "525be0a98f144ed2a352c7e5b15266d0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38797 } } peers { permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 } }
I20260926 06:26:40.915867 13911 ts_tablet_manager.cc:1428] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:26:40.916405 13893 heartbeater.cc:503] Master 127.2.39.254:38417 was elected leader, sending a full tablet report...
I20260926 06:26:40.917152 13916 raft_consensus.cc:348] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 } } peers { permanent_uuid: "525be0a98f144ed2a352c7e5b15266d0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38797 } } peers { permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 } }
I20260926 06:26:40.917366 13916 raft_consensus.cc:374] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:26:40.917411 13916 raft_consensus.cc:729] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0cae01b02541cc903c8c8106e557cb, State: Initialized, Role: FOLLOWER
I20260926 06:26:40.917560 13916 consensus_queue.cc:260] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 } } peers { permanent_uuid: "525be0a98f144ed2a352c7e5b15266d0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38797 } } peers { permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 } }
I20260926 06:26:40.917922 13621 heartbeater.cc:503] Master 127.2.39.254:38417 was elected leader, sending a full tablet report...
I20260926 06:26:40.917919 13916 ts_tablet_manager.cc:1428] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
Sep 26 06:26:40 dist-test-slave-fkbn krb5kdc[12933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:26:40.951097 13918 raft_consensus.cc:482] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:26:40.951212 13918 raft_consensus.cc:504] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 } } peers { permanent_uuid: "525be0a98f144ed2a352c7e5b15266d0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38797 } } peers { permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 } }
I20260926 06:26:40.951577 13918 leader_election.cc:288] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8ebf44ca3a2472fb83e7d185e5953ce (127.2.39.194:45289), 1e0cae01b02541cc903c8c8106e557cb (127.2.39.193:38881)
I20260926 06:26:40.955945 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.951805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:45289 (local address 127.2.39.195:49915)
0926 06:26:40.952345 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 06:26:40.952359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:26:40.952406 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:26:40.952818 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:26:40.952819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:26:40.952821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:26:40.952897 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.952899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.953820 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.953821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.954939 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.954942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.955684 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.955685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.955759 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.955763 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:26:40.955813 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:26:40.955867 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520}
I20260926 06:26:40.956663 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.952543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:49915 (local address 127.2.39.194:45289)
0926 06:26:40.952653 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:40.952655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:40.952663 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:26:40.952688 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:40.952689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:40.952699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:40.952743 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:26:40.952965 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.953669 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.955039 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.955569 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.955642 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.955651 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:40.956554 (+ 903us) server_negotiation.cc:299] Negotiation successful
0926 06:26:40.956596 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 06:26:40.956970 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723323a84fc4e5f8a26ce2bc302c624" candidate_uuid: "525be0a98f144ed2a352c7e5b15266d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" is_pre_election: true
I20260926 06:26:40.957118 13713 raft_consensus.cc:2454] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525be0a98f144ed2a352c7e5b15266d0 in term 0.
I20260926 06:26:40.957508 13782 leader_election.cc:302] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525be0a98f144ed2a352c7e5b15266d0, d8ebf44ca3a2472fb83e7d185e5953ce; no voters:
I20260926 06:26:40.957738 13918 raft_consensus.cc:2781] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:26:40.957798 13918 raft_consensus.cc:482] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:26:40.957820 13918 raft_consensus.cc:3037] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:26:40.958681 13918 raft_consensus.cc:504] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 } } peers { permanent_uuid: "525be0a98f144ed2a352c7e5b15266d0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38797 } } peers { permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 } }
I20260926 06:26:40.958803 13918 leader_election.cc:288] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [CANDIDATE]: Term 1 election: Requested vote from peers d8ebf44ca3a2472fb83e7d185e5953ce (127.2.39.194:45289), 1e0cae01b02541cc903c8c8106e557cb (127.2.39.193:38881)
I20260926 06:26:40.959095 13713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723323a84fc4e5f8a26ce2bc302c624" candidate_uuid: "525be0a98f144ed2a352c7e5b15266d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce"
I20260926 06:26:40.959162 13713 raft_consensus.cc:3037] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:26:40.960007 13713 raft_consensus.cc:2454] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525be0a98f144ed2a352c7e5b15266d0 in term 1.
I20260926 06:26:40.960214 13782 leader_election.cc:302] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 525be0a98f144ed2a352c7e5b15266d0, d8ebf44ca3a2472fb83e7d185e5953ce; no voters:
I20260926 06:26:40.960335 13918 raft_consensus.cc:2781] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:26:40.960479 13918 raft_consensus.cc:686] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 1 LEADER]: Becoming Leader. State: Replica: 525be0a98f144ed2a352c7e5b15266d0, State: Running, Role: LEADER
I20260926 06:26:40.960570 13918 consensus_queue.cc:237] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 } } peers { permanent_uuid: "525be0a98f144ed2a352c7e5b15266d0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38797 } } peers { permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 } }
I20260926 06:26:40.960623 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.952614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:42703 (local address 127.2.39.193:38881)
0926 06:26:40.952792 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:40.952793 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:40.956129 (+ 3336us) server_negotiation.cc:407] Connection header received
0926 06:26:40.956158 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:40.956159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:40.956172 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:40.956217 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:26:40.957759 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.958408 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.959597 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.960198 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.960261 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.960270 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:40.960510 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 06:26:40.960549 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260926 06:26:40.960770 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.951959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:38881 (local address 127.2.39.195:42703)
0926 06:26:40.956017 (+ 4058us) negotiation.cc:107] Waiting for socket to connect
0926 06:26:40.956028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:26:40.956072 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:26:40.957623 (+ 1551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:26:40.957624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:26:40.957626 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:26:40.957665 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.957666 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.958504 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.958505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.959500 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:26:40.959502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.960297 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.960299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:26:40.960375 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.960379 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:26:40.960429 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 06:26:40.960470 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4047,"thread_start_us":91,"threads_started":1}
I20260926 06:26:40.961406 13409 catalog_manager.cc:5495] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 525be0a98f144ed2a352c7e5b15266d0 (127.2.39.195). New cstate: current_term: 1 leader_uuid: "525be0a98f144ed2a352c7e5b15266d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "525be0a98f144ed2a352c7e5b15266d0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20260926 06:26:40.962694 13575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723323a84fc4e5f8a26ce2bc302c624" candidate_uuid: "525be0a98f144ed2a352c7e5b15266d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e0cae01b02541cc903c8c8106e557cb" is_pre_election: true
I20260926 06:26:40.962850 13575 raft_consensus.cc:2454] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 525be0a98f144ed2a352c7e5b15266d0 in term 0.
I20260926 06:26:40.962949 13574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0723323a84fc4e5f8a26ce2bc302c624" candidate_uuid: "525be0a98f144ed2a352c7e5b15266d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e0cae01b02541cc903c8c8106e557cb"
I20260926 06:26:40.962994 13574 raft_consensus.cc:3037] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:26:40.963796 13574 raft_consensus.cc:2454] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 525be0a98f144ed2a352c7e5b15266d0 in term 1.
I20260926 06:26:40.980243 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:40.976760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.2.39.195:38797)
0926 06:26:40.976802 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:26:40.976804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:26:40.976832 (+ 28us) server_negotiation.cc:407] Connection header received
0926 06:26:40.976966 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:26:40.976967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:26:40.976988 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:26:40.977081 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:26:40.977272 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.978025 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.979262 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:26:40.979573 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:26:40.979658 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:26:40.979853 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:26:40.979982 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:26:40.980077 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 06:26:40.980125 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 06:26:40.980147 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:26:40.983019 13713 raft_consensus.cc:1261] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce [term 1 FOLLOWER]: Refusing update from remote peer 525be0a98f144ed2a352c7e5b15266d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:26:40.983502 13574 raft_consensus.cc:1261] T 0723323a84fc4e5f8a26ce2bc302c624 P 1e0cae01b02541cc903c8c8106e557cb [term 1 FOLLOWER]: Refusing update from remote peer 525be0a98f144ed2a352c7e5b15266d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:26:40.984026 13918 consensus_queue.cc:1035] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8ebf44ca3a2472fb83e7d185e5953ce" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:26:40.985085 13918 consensus_queue.cc:1035] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e0cae01b02541cc903c8c8106e557cb" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:26:40.991165 13924 mvcc.cc:205] Tried to move back new op lower bound from 7333494788022497280 to 7333494787935113216. Current Snapshot: MvccSnapshot[applied={T|T < 7333494788022497280}]
I20260926 06:26:40.993036 13925 mvcc.cc:205] Tried to move back new op lower bound from 7333494788022497280 to 7333494787935113216. Current Snapshot: MvccSnapshot[applied={T|T < 7333494788022497280}]
I20260926 06:26:40.993593 13927 mvcc.cc:205] Tried to move back new op lower bound from 7333494788022497280 to 7333494787935113216. Current Snapshot: MvccSnapshot[applied={T|T < 7333494788022497280}]
I20260926 06:26:40.995918 13409 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:26:40.996007 13409 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:26:40.996752 13409 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0978ff28732d4f7f8b073b05e2b78b40: Sending DeleteTablet for 3 replicas of tablet 0723323a84fc4e5f8a26ce2bc302c624
I20260926 06:26:40.997692 13674 tablet_service.cc:1514] Processing DeleteTablet for tablet 0723323a84fc4e5f8a26ce2bc302c624 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:26:40 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:42182
I20260926 06:26:40.997895 13552 tablet_service.cc:1514] Processing DeleteTablet for tablet 0723323a84fc4e5f8a26ce2bc302c624 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:26:40 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:37532
I20260926 06:26:40.998064 13826 tablet_service.cc:1514] Processing DeleteTablet for tablet 0723323a84fc4e5f8a26ce2bc302c624 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:26:40 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:50642
I20260926 06:26:40.998716 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 13490
I20260926 06:26:41.009692 13933 tablet_replica.cc:330] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: stopping tablet replica
I20260926 06:26:41.013748 13935 tablet_replica.cc:330] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: stopping tablet replica
I20260926 06:26:41.014045 13935 raft_consensus.cc:2229] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:26:41.014225 13933 raft_consensus.cc:2229] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:26:41.014427 13935 raft_consensus.cc:2258] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:26:41.014588 13933 raft_consensus.cc:2258] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:26:41.015401 13935 ts_tablet_manager.cc:1905] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:26:41.015921 13933 ts_tablet_manager.cc:1905] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:26:41.019028 13935 ts_tablet_manager.cc:1918] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:26:41.019378 13933 ts_tablet_manager.cc:1918] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:26:41.019500 13933 log.cc:1196] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/0723323a84fc4e5f8a26ce2bc302c624
I20260926 06:26:41.019822 13933 ts_tablet_manager.cc:1939] T 0723323a84fc4e5f8a26ce2bc302c624 P d8ebf44ca3a2472fb83e7d185e5953ce: Deleting consensus metadata
I20260926 06:26:41.020113 13935 log.cc:1196] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/0723323a84fc4e5f8a26ce2bc302c624
I20260926 06:26:41.020530 13935 ts_tablet_manager.cc:1939] T 0723323a84fc4e5f8a26ce2bc302c624 P 525be0a98f144ed2a352c7e5b15266d0: Deleting consensus metadata
I20260926 06:26:41.021062 13395 catalog_manager.cc:4841] TS d8ebf44ca3a2472fb83e7d185e5953ce (127.2.39.194:45289): tablet 0723323a84fc4e5f8a26ce2bc302c624 (table test-table [id=8b2a361fa7cc45fe84f090e218b7e5f2]) successfully deleted
I20260926 06:26:41.021622 13393 catalog_manager.cc:4841] TS 525be0a98f144ed2a352c7e5b15266d0 (127.2.39.195:38797): tablet 0723323a84fc4e5f8a26ce2bc302c624 (table test-table [id=8b2a361fa7cc45fe84f090e218b7e5f2]) successfully deleted
W20260926 06:26:41.021983 13395 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:38881 (error 108)
I20260926 06:26:41.029692 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 13627
I20260926 06:26:41.031186 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:26:41.030631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:38881 (local address 127.0.0.1:37542)
0926 06:26:41.031092 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 06:26:41.031149 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431}
W20260926 06:26:41.031368 13395 catalog_manager.cc:4584] TS 1e0cae01b02541cc903c8c8106e557cb (127.2.39.193:38881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0723323a84fc4e5f8a26ce2bc302c624: Network error: Client connection negotiation failed: client connection to 127.2.39.193:38881: connect: Connection refused (error 111)
I20260926 06:26:41.037014 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 13764
I20260926 06:26:41.043533 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 13377
I20260926 06:26:41.050094 2207 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 06:26:41.394488 2207 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 06:26:41.394609 2207 mini_ranger.cc:66] Stopping Ranger...
I20260926 06:26:41.739151 2207 mini_ranger.cc:68] Stopped Ranger
2026-09-26 06:26:41.739 UTC [12963] LOG: received smart shutdown request
2026-09-26 06:26:41.743 UTC [12963] LOG: background worker "logical replication launcher" (PID 12972) exited with exit code 1
2026-09-26 06:26:41.743 UTC [12967] LOG: shutting down
2026-09-26 06:26:41.919 UTC [12963] LOG: database system is shut down
2026-09-26T06:26:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45059 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:26:42 dist-test-slave-fkbn krb5kdc[13945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:26:42 dist-test-slave-fkbn krb5kdc[13945](info): set up 2 sockets
Sep 26 06:26:42 dist-test-slave-fkbn krb5kdc[13945](info): commencing operation
krb5kdc: starting...
W20260926 06:26:43.069651 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:26:43 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:26:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:26:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.39.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.39.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/rangeradmin_127.2.39.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/rangeradmin_127.2.39.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.39.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.39.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/rangerlookup_127.2.39.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/rangerlookup_127.2.39.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.39.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.39.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/HTTP_127.2.39.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/HTTP_127.2.39.212@KRBTEST.COM.keytab.
I20260926 06:26:43.235793 2207 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOgNaKF/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/postgres -l logfile start
2026-09-26 06:26:45.716 UTC [13975] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 06:26:45.716 UTC [13975] LOG: listening on IPv4 address "127.2.39.212", port 37121
2026-09-26 06:26:45.719 UTC [13975] LOG: listening on Unix socket "/tmp/.s.PGSQL.37121"
2026-09-26 06:26:45.740 UTC [13978] LOG: database system was shut down at 2026-09-26 06:26:44 UTC
2026-09-26 06:26:45.744 UTC [13975] LOG: database system is ready to accept connections
I20260926 06:26:46.693352 2207 mini_postgres.cc:96] Postgres bound to 37121
2026-09-26 06:26:46.699 UTC [13986] FATAL: database "slave" does not exist
127.2.39.212:37121 - accepting connections
I20260926 06:26:46.699985 2207 mini_ranger.cc:161] Starting Ranger...
I20260926 06:26:46.714321 2207 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 06:26:47.196859 2207 mini_ranger.cc:87] Created ranger Postgres database
I20260926 06:26:47.196949 2207 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-admin
2026-09-26 06:26:47,471 [I] DB FLAVOR :POSTGRES
2026-09-26 06:26:47,472 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:26:47,472 [I] Checking connection..
2026-09-26 06:26:47,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 06:26:47,717 [I] Checking connection passed.
2026-09-26 06:26:47,718 [I] --------- Verifying version history table ---------
2026-09-26 06:26:47,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:26:47,949 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:26:47,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:26:48,197 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 06:26:48,197 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 06:26:48,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 06:26:48.452 UTC [14050] WARNING: there is no transaction in progress
2026-09-26 06:26:48,458 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 06:26:48,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 06:26:48,694 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 06:26:48,694 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:26:48,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 06:26:48,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 06:26:49,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-fkbn.c.gcp-upstream.internal','N') ;"
2026-09-26 06:26:49,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:26:49,696 [I] Table x_portal_user does not exist in database ranger
2026-09-26 06:26:49,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:26:49,952 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 06:26:49,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:26:50,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:26:50,442 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 06:26:50,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 06:26:51.367 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.384 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.401 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.417 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.434 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.454 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.511 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.518 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.533 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.544 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.554 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.569 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.580 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.586 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.594 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:51.601 UTC [14158] WARNING: there is no transaction in progress
2026-09-26 06:26:52,263 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 06:26:52,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 06:26:52,502 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 06:26:52,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 06:26:52,730 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 06:26:52,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 06:26:52,993 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 06:26:52,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 06:26:53,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 06:26:53,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-fkbn.c.gcp-upstream.internal';"
2026-09-26 06:26:53,725 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 06:26:53,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 06:26:53,974 [I] DB_PATCHES have already been applied
I20260926 06:26:53.979879 2207 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOgNaKF/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 06:26:53.979964 2207 mini_ranger.cc:193] Using host: 127.2.39.212
I20260926 06:26:53.984390 2207 mini_ranger.cc:227] Ranger admin URL: http://127.2.39.212:35869
Sep 26, 2026 6:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 6:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 6:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 6:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 6:26:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 06:26:55.109511 2207 mini_ranger.cc:160] Time spent starting Ranger: real 8.410s user 0.003s sys 0.006s
Sep 26, 2026 6:27:15 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 6:27:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 6:27:15 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 6:27:15 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 6:27:16 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 06:27:17.015605 2207 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.39.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.39.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/rangerkms_127.2.39.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.39.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/rangerkms_127.2.39.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.39.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.39.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.39.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/HTTP_127.2.39.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.39.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/HTTP_127.2.39.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 06:27:17 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404037, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 06:27:17.102972 2207 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 06:27:17.112437 2207 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 06:27:18.046029 2207 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 06:27:18.046100 2207 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-kms
I20260926 06:27:18.046114 2207 mini_ranger_kms.cc:227] Using postgres at 127.2.39.212:37121
2026-09-26 06:27:18,235 [I] DB FLAVOR :POSTGRES
2026-09-26 06:27:18,235 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 06:27:18,235 [I] Checking connection
2026-09-26 06:27:18,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 06:27:18,463 [I] connection success
2026-09-26 06:27:18,463 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 06:27:18,463 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 06:27:18,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 06:27:18,710 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 06:27:18,710 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 06:27:18,710 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 06:27:18,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.39.212:37121/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 06:27:18,977 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 06:27:19.448143 2207 mini_ranger_kms.cc:310] Created kms service
I20260926 06:27:19.640496 2207 mini_ranger_kms.cc:326] Created kudu user
I20260926 06:27:19.683153 2207 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 06:27:19 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404037, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:20.080389 2207 mini_ranger_kms.cc:384] Added ranger policy
I20260926 06:27:20.080510 2207 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOgNaKF/build/release/bin/postgresql.jar:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOgNaKF/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 06:27:20.080530 2207 mini_ranger_kms.cc:240] Using host: 127.2.39.212
I20260926 06:27:20.084192 2207 mini_ranger_kms.cc:276] Ranger KMS PID: 14338
I20260926 06:27:20.084283 2207 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.39.212:40061
Sep 26, 2026 6:27:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 6:27:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 6:27:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 6:27:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 6:27:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 6:27:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 06:27:22.381239 2207 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.278s user 0.000s sys 0.159s
I20260926 06:27:22.381346 2207 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 06:27:22.381397 2207 mini_ranger_kms.cc:401] 127.2.39.212:40061/kms/v1/keys
Sep 26 06:27:22 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790404042, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.39.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 06:27:23 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404042, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.39.212@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
Sep 26 06:27:23 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404042, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.39.212@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
Sep 26 06:27:25 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:27:25.624109 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:41391
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:41391
--ranger_config_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:40061/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:25.704121 14389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:25.704320 14389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:25.704341 14389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:25.705488 14389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:27:25.705526 14389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:25.705540 14389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:25.705552 14389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:27:25.705579 14389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:27:25.707049 14389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.39.212:40061/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:41391
--ranger_config_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:25.707327 14389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:25.707561 14389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:25.712734 14395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:25.712754 14394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:25.712755 14397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:25.712954 14389 server_base.cc:1034] running on GCE node
I20260926 06:27:25.713667 14389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:25.713956 14389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:25.715131 14389 hybrid_clock.cc:648] HybridClock initialized: now 1790404045715118 us; error 31 us; skew 500 ppm
Sep 26 06:27:25 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:25.723104 14389 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:27:25.723613 14389 webserver.cc:460] Webserver started at http://127.2.39.254:33645/ using document root <none> and password file <none>
I20260926 06:27:25.723842 14389 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:25.723894 14389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:25.724030 14389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:27:25 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:25.773188 14389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "7beeaf8688cf404285ebfaf198cf1d49"
format_stamp: "Formatted at 2026-09-26 06:27:25 on dist-test-slave-fkbn"
server_key: "d695e6a4ca0a9765da069364c2c1ba21"
server_key_iv: "42743370427f610d6c79a5d938c6f68b"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:25.773628 14389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "7beeaf8688cf404285ebfaf198cf1d49"
format_stamp: "Formatted at 2026-09-26 06:27:25 on dist-test-slave-fkbn"
server_key: "d695e6a4ca0a9765da069364c2c1ba21"
server_key_iv: "42743370427f610d6c79a5d938c6f68b"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:25.775113 14389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:27:25.787135 14404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:25.787365 14389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:25.787465 14389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "7beeaf8688cf404285ebfaf198cf1d49"
format_stamp: "Formatted at 2026-09-26 06:27:25 on dist-test-slave-fkbn"
server_key: "d695e6a4ca0a9765da069364c2c1ba21"
server_key_iv: "42743370427f610d6c79a5d938c6f68b"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:25.787580 14389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:25.796764 14389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:25.798748 14389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:25.798895 14389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:25.803743 14389 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:41391
I20260926 06:27:25.803826 14456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:41391 every 8 connection(s)
I20260926 06:27:25.804184 14389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:27:25.804891 14457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:25.807238 14457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: Bootstrap starting.
I20260926 06:27:25.807817 14457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:25.808108 14457 log.cc:824] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:25.808777 14457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: No bootstrap required, opened a new log
I20260926 06:27:25.809441 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 14389
I20260926 06:27:25.809541 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:27:25.809792 2207 external_mini_cluster.cc:1428] Setting key fcbfcc8ee020bd4ff02cb94ee8eb900b
I20260926 06:27:25.811321 14457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } }
I20260926 06:27:25.811482 14457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:25.811569 14457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7beeaf8688cf404285ebfaf198cf1d49, State: Initialized, Role: FOLLOWER
I20260926 06:27:25.811741 14457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } }
I20260926 06:27:25.811837 14457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:27:25.811892 14457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:27:25.811961 14457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:25.812505 14457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } }
I20260926 06:27:25.812633 14457 leader_election.cc:302] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7beeaf8688cf404285ebfaf198cf1d49; no voters:
I20260926 06:27:25.812793 14457 leader_election.cc:288] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:27:25.812840 14462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:27:25.813045 14462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 1 LEADER]: Becoming Leader. State: Replica: 7beeaf8688cf404285ebfaf198cf1d49, State: Running, Role: LEADER
I20260926 06:27:25.813117 14457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:27:25.813170 14462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } }
I20260926 06:27:25.813535 14462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7beeaf8688cf404285ebfaf198cf1d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } } }
I20260926 06:27:25.813719 14462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:25.813901 14462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7beeaf8688cf404285ebfaf198cf1d49. Latest consensus state: current_term: 1 leader_uuid: "7beeaf8688cf404285ebfaf198cf1d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } } }
I20260926 06:27:25.814002 14462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:25.814168 14471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:27:25.814663 14471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:27:25.814904 14457 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 26 06:27:25 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:25.825096 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:25.810323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60224 (local address 127.2.39.254:41391)
0926 06:27:25.810564 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:25.810574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:25.810606 (+ 32us) server_negotiation.cc:407] Connection header received
0926 06:27:25.811331 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:25.811340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:25.811419 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:25.812377 (+ 958us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:25.812495 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:25.813988 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:25.815114 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:25.817776 (+ 2662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:25.819637 (+ 1861us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:25.820820 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:25.820824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:25.820825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:25.820836 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:25.822986 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:25.823442 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:25.823445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:25.823446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:25.823552 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:25.823766 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:25.823767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:25.823768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:25.824061 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:25.824570 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:25.824645 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 06:27:25.824793 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":69,"threads_started":1}
I20260926 06:27:25.826146 14471 catalog_manager.cc:1347] Generated new cluster ID: 562498fefff144058d7cf0522decb8e3
I20260926 06:27:25.826195 14471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:27:25.845345 14471 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:27:25.846378 14471 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:27:25.850327 14471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: Generated new TSK 0
I20260926 06:27:25.850579 14471 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:27:26.516556 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:40061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41391
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:26.602756 14502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:26.602967 14502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:26.603006 14502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:26.604205 14502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:26.604272 14502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:26.604316 14502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:27:26.605819 14502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.39.212:40061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:26.606070 14502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:26.606303 14502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:26.611193 14507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:26.611323 14508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:26.611202 14510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:26.612531 14502 server_base.cc:1034] running on GCE node
I20260926 06:27:26.612782 14502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:26.613080 14502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:26.614296 14502 hybrid_clock.cc:648] HybridClock initialized: now 1790404046614306 us; error 56 us; skew 500 ppm
Sep 26 06:27:26 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:26.616842 14502 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:27:26.617334 14502 webserver.cc:460] Webserver started at http://127.2.39.193:35539/ using document root <none> and password file <none>
I20260926 06:27:26.617650 14502 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:26.617713 14502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:26.617853 14502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:27:26 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:26.629060 14502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "85b512f6af774e2799456bc7b6b2a457"
format_stamp: "Formatted at 2026-09-26 06:27:26 on dist-test-slave-fkbn"
server_key: "913d20327b07834bf48904e0c012c719"
server_key_iv: "05637289af8eeb266dd87f230d780b9b"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:26.629552 14502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "85b512f6af774e2799456bc7b6b2a457"
format_stamp: "Formatted at 2026-09-26 06:27:26 on dist-test-slave-fkbn"
server_key: "913d20327b07834bf48904e0c012c719"
server_key_iv: "05637289af8eeb266dd87f230d780b9b"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:26.631388 14502 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260926 06:27:26.646399 14517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:26.646616 14502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:27:26.646719 14502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "85b512f6af774e2799456bc7b6b2a457"
format_stamp: "Formatted at 2026-09-26 06:27:26 on dist-test-slave-fkbn"
server_key: "913d20327b07834bf48904e0c012c719"
server_key_iv: "05637289af8eeb266dd87f230d780b9b"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:26.646816 14502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:26.659449 14502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:26.661800 14502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:26.661947 14502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:26.662200 14502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:26.662616 14502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:26.662706 14502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:26.662806 14502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:26.662880 14502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:26.670739 14502 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:45675
I20260926 06:27:26.670795 14630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:45675 every 8 connection(s)
I20260926 06:27:26.671173 14502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:27:26.673501 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 14502
I20260926 06:27:26.673619 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:27:26.673812 2207 external_mini_cluster.cc:1428] Setting key bb170a18512da961dea32ecaea38ed33
Sep 26 06:27:26 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:26.683643 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:26.671933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:52279 (local address 127.2.39.254:41391)
0926 06:27:26.672731 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:26.672732 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:26.672739 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:27:26.672762 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:26.672762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:26.672771 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:26.672857 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:26.673458 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:26.675161 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:26.676725 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:26.677802 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:26.677899 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:26.680324 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:26.680326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:26.680326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:26.680346 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:26.682167 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:26.682655 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:26.682656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:26.682657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:26.682721 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:26.682984 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:26.682985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:26.682985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:26.683103 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:26.683183 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:26.683463 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 06:27:26.683549 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":85,"threads_started":1}
I20260926 06:27:26.683954 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:26.671905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41391 (local address 127.2.39.193:52279)
0926 06:27:26.672084 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:26.672127 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:26.672591 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:26.672943 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:26.672947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:26.673129 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:26.673335 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:26.673341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:26.675269 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:26.675270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:26.676616 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:26.676619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:26.677931 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:26.677933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:26.678059 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:26.678611 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:26.678621 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:26.680217 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:26.682258 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:26.682264 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:26.682266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:26.682569 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:26.682802 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:26.682803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:26.682803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:26.682910 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:26.683187 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:26.683193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:26.683324 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 06:27:26.683431 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 06:27:26.683566 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260926 06:27:26.684445 14631 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41391
I20260926 06:27:26.684541 14631 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:26.684736 14631 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
I20260926 06:27:26.685254 14421 ts_manager.cc:194] Registered new tserver with Master: 85b512f6af774e2799456bc7b6b2a457 (127.2.39.193:45675)
I20260926 06:27:26.686142 14421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:52279
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:27:26.714289 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:40061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41391
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:26.813459 14639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:26.813856 14639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:26.813954 14639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:26.815716 14639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:26.815821 14639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:26.815918 14639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:27:26.818097 14639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.39.212:40061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:26.818529 14639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:26.818821 14639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:26.825213 14647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:26.825553 14646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:26.826756 14639 server_base.cc:1034] running on GCE node
W20260926 06:27:26.826990 14649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:26.827240 14639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:26.827495 14639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:26.828668 14639 hybrid_clock.cc:648] HybridClock initialized: now 1790404046828640 us; error 40 us; skew 500 ppm
Sep 26 06:27:26 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:26.831890 14639 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:27:26.832415 14639 webserver.cc:460] Webserver started at http://127.2.39.194:46651/ using document root <none> and password file <none>
I20260926 06:27:26.832742 14639 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:26.832808 14639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:26.832934 14639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:27:26 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:26.846411 14639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "d961e52de4e14707bdfe1066fd942a83"
format_stamp: "Formatted at 2026-09-26 06:27:26 on dist-test-slave-fkbn"
server_key: "cbcffe554019822f35a3fea3898d87e8"
server_key_iv: "e3116e6a99f2e7503b8e40156507a669"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:26.848335 14639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "d961e52de4e14707bdfe1066fd942a83"
format_stamp: "Formatted at 2026-09-26 06:27:26 on dist-test-slave-fkbn"
server_key: "cbcffe554019822f35a3fea3898d87e8"
server_key_iv: "e3116e6a99f2e7503b8e40156507a669"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:26.853953 14639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 06:27:26.869726 14656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:26.869997 14639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 06:27:26.870087 14639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "d961e52de4e14707bdfe1066fd942a83"
format_stamp: "Formatted at 2026-09-26 06:27:26 on dist-test-slave-fkbn"
server_key: "cbcffe554019822f35a3fea3898d87e8"
server_key_iv: "e3116e6a99f2e7503b8e40156507a669"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:26.870180 14639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:26.887627 14639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:26.890165 14639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:26.890349 14639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:26.890616 14639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Sep 26 06:27:26 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790404046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:26.891000 14639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:26.891098 14639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:26.891171 14639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:26.891235 14639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:26.897408 14639 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:46515
I20260926 06:27:26.897903 14639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:27:26.899835 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 14639
I20260926 06:27:26.899937 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:27:26.900128 2207 external_mini_cluster.cc:1428] Setting key e1e5d47f6a33a8051f89d489a3a7adc2
I20260926 06:27:26.902024 14769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:46515 every 8 connection(s)
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Sep 26 06:27:26 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:26.921753 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:26.902891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:38715 (local address 127.2.39.254:41391)
0926 06:27:26.902938 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:26.902940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:26.911176 (+ 8236us) server_negotiation.cc:407] Connection header received
0926 06:27:26.911871 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:26.911873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:26.911891 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:26.911965 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:26.912792 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:26.913628 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:26.915332 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:26.915689 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:26.915739 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:26.918432 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:26.918433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:26.918434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:26.918441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:26.920169 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:26.920648 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:26.920649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:26.920650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:26.920706 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:26.920998 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:26.920999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:26.921000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:26.921130 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:26.921174 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:26.921546 (+ 372us) server_negotiation.cc:299] Negotiation successful
0926 06:27:26.921657 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:27:26.922253 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:26.902615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41391 (local address 127.2.39.194:38715)
0926 06:27:26.910999 (+ 8384us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:26.911060 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:26.911745 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:26.912116 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:26.912121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:26.912368 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:26.912668 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:26.912675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:26.913750 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:26.913750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:26.915231 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:26.915233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:26.915880 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:26.915883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:26.916023 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:26.916661 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:26.916671 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:26.918319 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:26.920266 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:26.920273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:26.920276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:26.920558 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:26.920776 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:26.920777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:26.920777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:26.920915 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:26.921221 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:26.921227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:26.921382 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 06:27:26.921862 (+ 480us) client_negotiation.cc:240] Negotiation successful
0926 06:27:26.922023 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8325,"thread_start_us":86,"threads_started":1}
I20260926 06:27:26.922808 14770 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41391
I20260926 06:27:26.922914 14770 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:26.923108 14770 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
I20260926 06:27:26.924808 14421 ts_manager.cc:194] Registered new tserver with Master: d961e52de4e14707bdfe1066fd942a83 (127.2.39.194:46515)
I20260926 06:27:26.925307 14421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:38715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:27:26.945436 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:40061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41391
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:27.048399 14775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:27.048679 14775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:27.048714 14775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:27.050369 14775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:27.050462 14775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:27.050531 14775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:27:27.052634 14775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.39.212:40061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:27.053016 14775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:27.053287 14775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:27.059778 14782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:27.059991 14784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:27.060233 14781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:27.061496 14775 server_base.cc:1034] running on GCE node
I20260926 06:27:27.061888 14775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:27.062234 14775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:27.063374 14775 hybrid_clock.cc:648] HybridClock initialized: now 1790404047063349 us; error 44 us; skew 500 ppm
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:27.066694 14775 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:27:27.067250 14775 webserver.cc:460] Webserver started at http://127.2.39.195:38729/ using document root <none> and password file <none>
I20260926 06:27:27.067524 14775 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:27.067590 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:27.067809 14775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:27.084177 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "7b1f5a243d0e4f59993fae7ae46d1100"
format_stamp: "Formatted at 2026-09-26 06:27:27 on dist-test-slave-fkbn"
server_key: "8b0a89a118fceb57e07e3ae0c03e94d2"
server_key_iv: "a3aebc0145a26211ef014a74c6fdaba8"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:27.084842 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "7b1f5a243d0e4f59993fae7ae46d1100"
format_stamp: "Formatted at 2026-09-26 06:27:27 on dist-test-slave-fkbn"
server_key: "8b0a89a118fceb57e07e3ae0c03e94d2"
server_key_iv: "a3aebc0145a26211ef014a74c6fdaba8"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:27.093976 14775 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.000s
I20260926 06:27:27.107368 14791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.107666 14775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:27.107784 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "7b1f5a243d0e4f59993fae7ae46d1100"
format_stamp: "Formatted at 2026-09-26 06:27:27 on dist-test-slave-fkbn"
server_key: "8b0a89a118fceb57e07e3ae0c03e94d2"
server_key_iv: "a3aebc0145a26211ef014a74c6fdaba8"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:27.107893 14775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:27.123659 14775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:27.127498 14775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:27.127713 14775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:27.128022 14775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:27.128473 14775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:27.128567 14775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.128646 14775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:27.128711 14775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.138166 14775 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:41971
I20260926 06:27:27.138674 14775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:27:27.142591 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 14775
I20260926 06:27:27.142688 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:27:27.142895 2207 external_mini_cluster.cc:1428] Setting key a120a38b32d6c17dca5410caea14bef8
I20260926 06:27:27.146190 14904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:41971 every 8 connection(s)
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:27.159498 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:27.147411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:53417 (local address 127.2.39.254:41391)
0926 06:27:27.147468 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:27.147470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:27.148585 (+ 1115us) server_negotiation.cc:407] Connection header received
0926 06:27:27.149332 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:27.149333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:27.149346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:27.149380 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:27.150338 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.151126 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.152781 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.153105 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.153150 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:27.156002 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:27.156003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:27.156003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:27.156011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:27.157857 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:27.158360 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:27.158361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:27.158362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:27.158427 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:27.158726 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:27.158727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:27.158728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:27.158872 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:27.158958 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:27.159304 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 06:27:27.159401 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:27:27.159657 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:27.147070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41391 (local address 127.2.39.195:53417)
0926 06:27:27.148444 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:27.148498 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:27.149204 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:27.149593 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:27.149598 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:27.149902 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:27.150229 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:27.150235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.151246 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.151247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:27.152682 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:27.152685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.153256 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.153258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:27.153402 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:27.154049 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:27.154059 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:27.155895 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:27.157952 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:27.157958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:27.157961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:27.158262 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:27.158503 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:27.158504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:27.158504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:27.158641 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:27.158958 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:27.158964 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:27.159134 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 06:27:27.159255 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 06:27:27.159415 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1315,"thread_start_us":105,"threads_started":1}
I20260926 06:27:27.160295 14905 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41391
I20260926 06:27:27.160403 14905 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:27.160601 14905 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
I20260926 06:27:27.160995 14421 ts_manager.cc:194] Registered new tserver with Master: 7b1f5a243d0e4f59993fae7ae46d1100 (127.2.39.195:41971)
I20260926 06:27:27.161468 14421 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:53417
I20260926 06:27:27.165462 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:27:27.166311 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 14502
I20260926 06:27:27.171751 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 14639
I20260926 06:27:27.176810 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 14775
I20260926 06:27:27.181380 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 14389
I20260926 06:27:27.186946 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:41391
--webserver_interface=127.2.39.254
--webserver_port=33645
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:41391
--ranger_config_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:40061/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.39.212:40061/kms with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:27.267196 14908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:27.267388 14908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:27.267408 14908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:27.268579 14908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:27:27.268621 14908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:27.268635 14908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:27.268646 14908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:27:27.268658 14908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:27:27.270175 14908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.39.212:40061/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:41391
--ranger_config_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=33645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:27.270351 14908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:27.270505 14908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:27.275202 14914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:27.275279 14913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:27.275213 14916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:27.275331 14908 server_base.cc:1034] running on GCE node
I20260926 06:27:27.275616 14908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:27.275910 14908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:27.277082 14908 hybrid_clock.cc:648] HybridClock initialized: now 1790404047277077 us; error 37 us; skew 500 ppm
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:27.279623 14908 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:27:27.280169 14908 webserver.cc:460] Webserver started at http://127.2.39.254:33645/ using document root <none> and password file <none>
I20260926 06:27:27.280438 14908 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:27.280550 14908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:27.327569 14908 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:27.328583 14924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.328799 14908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:27.328903 14908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "7beeaf8688cf404285ebfaf198cf1d49"
format_stamp: "Formatted at 2026-09-26 06:27:25 on dist-test-slave-fkbn"
server_key: "d695e6a4ca0a9765da069364c2c1ba21"
server_key_iv: "42743370427f610d6c79a5d938c6f68b"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:27.329380 14908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:27.350922 14908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:27.352933 14908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:27.353082 14908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:27.358130 14908 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:41391
I20260926 06:27:27.358160 14976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:41391 every 8 connection(s)
I20260926 06:27:27.358623 14908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:27:27.359815 14977 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:27:27.360443 14977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: Bootstrap starting.
I20260926 06:27:27.362813 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 14908
I20260926 06:27:27.363101 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:40061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:45675
--local_ip_for_outbound_sockets=127.2.39.193
--tserver_master_addrs=127.2.39.254:41391
--webserver_port=35539
--webserver_interface=127.2.39.193
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.39.212:40061/kms with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:27:27.368956 14977 log.cc:824] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:27.370714 14977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:27:27.371029 14977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: Bootstrap complete.
I20260926 06:27:27.373378 14977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } }
I20260926 06:27:27.373677 14977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7beeaf8688cf404285ebfaf198cf1d49, State: Initialized, Role: FOLLOWER
I20260926 06:27:27.373888 14977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } }
I20260926 06:27:27.373986 14977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:27:27.374059 14977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:27:27.374114 14977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:27:27.375106 14977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } }
I20260926 06:27:27.375224 14977 leader_election.cc:302] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7beeaf8688cf404285ebfaf198cf1d49; no voters:
I20260926 06:27:27.375409 14977 leader_election.cc:288] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:27:27.375442 14981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:27:27.375649 14981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [term 2 LEADER]: Becoming Leader. State: Replica: 7beeaf8688cf404285ebfaf198cf1d49, State: Running, Role: LEADER
I20260926 06:27:27.375753 14977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:27:27.375790 14981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } }
I20260926 06:27:27.376109 14981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7beeaf8688cf404285ebfaf198cf1d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } } }
I20260926 06:27:27.376246 14981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:27.376427 14981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7beeaf8688cf404285ebfaf198cf1d49. Latest consensus state: current_term: 2 leader_uuid: "7beeaf8688cf404285ebfaf198cf1d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7beeaf8688cf404285ebfaf198cf1d49" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41391 } } }
I20260926 06:27:27.376556 14981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:27.376703 14989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:27:27.377239 14989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:27:27.377418 14989 catalog_manager.cc:1259] Loaded cluster ID: 562498fefff144058d7cf0522decb8e3
I20260926 06:27:27.377497 14989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:27:27.377676 14977 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 06:27:27.381348 14989 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:27:27.381956 14989 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: Loaded TSK: 0
I20260926 06:27:27.382314 14989 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:27:27.487349 14979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:27.487727 14979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:27.487843 14979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:27.489542 14979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:27.489697 14979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:27.489791 14979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:27:27.492014 14979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.39.212:40061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:45675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=35539
--tserver_master_addrs=127.2.39.254:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:27.492283 14979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:27.492539 14979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:27.498831 15015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:27.498863 15017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:27.499118 15014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:27.499375 14979 server_base.cc:1034] running on GCE node
I20260926 06:27:27.499620 14979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:27.499902 14979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:27.501060 14979 hybrid_clock.cc:648] HybridClock initialized: now 1790404047501039 us; error 39 us; skew 500 ppm
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:27.504038 14979 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:27:27.504621 14979 webserver.cc:460] Webserver started at http://127.2.39.193:35539/ using document root <none> and password file <none>
I20260926 06:27:27.504944 14979 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:27.505072 14979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:27.525113 14979 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260926 06:27:27.526196 15025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.526413 14979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:27:27.526499 14979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "85b512f6af774e2799456bc7b6b2a457"
format_stamp: "Formatted at 2026-09-26 06:27:26 on dist-test-slave-fkbn"
server_key: "913d20327b07834bf48904e0c012c719"
server_key_iv: "05637289af8eeb266dd87f230d780b9b"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:27.526947 14979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:27.554932 14979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:27.557552 14979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:27.557718 14979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:27.557919 14979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:27.558254 14979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:27.558302 14979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.558336 14979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:27.558353 14979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.564967 14979 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:45675
I20260926 06:27:27.565021 15138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:45675 every 8 connection(s)
I20260926 06:27:27.565456 14979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:27.573859 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 14979
I20260926 06:27:27.574143 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:40061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:46515
--local_ip_for_outbound_sockets=127.2.39.194
--tserver_master_addrs=127.2.39.254:41391
--webserver_port=46651
--webserver_interface=127.2.39.194
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.39.212:40061/kms with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:27:27.577955 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:27.566249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41391 (local address 127.2.39.193:42109)
0926 06:27:27.566463 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:27.566512 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:27.566986 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:27.567521 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:27.567524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:27.567696 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:27.567879 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:27.567883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.569525 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.569526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:27.570384 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:27.570386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.571280 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.571282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:27.571432 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:27.572087 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:27.572095 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:27.573450 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:27.575918 (+ 2468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:27.575924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:27.575926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:27.576246 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:27.576537 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:27.576538 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:27.576538 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:27.576710 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:27.577208 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:27.577214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:27.577372 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 06:27:27.577475 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 06:27:27.577672 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
I20260926 06:27:27.578498 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:27.566349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:42109 (local address 127.2.39.254:41391)
0926 06:27:27.566574 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:27.566585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:27.566630 (+ 45us) server_negotiation.cc:407] Connection header received
0926 06:27:27.567246 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:27.567254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:27.567312 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:27.567478 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:27.568012 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.569419 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.570487 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.571122 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.571292 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:27.573586 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:27.573592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:27.573592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:27.573604 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:27.575831 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:27.576339 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:27.576342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:27.576343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:27.576451 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:27.576784 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:27.576784 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:27.576785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:27.577098 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:27.577179 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:27.578149 (+ 970us) server_negotiation.cc:299] Negotiation successful
0926 06:27:27.578264 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260926 06:27:27.583617 15139 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41391
I20260926 06:27:27.583742 15139 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:27.583952 15139 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
W20260926 06:27:27.584895 15139 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:41391 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 06:27:27.584981 15139 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:27.585057 15139 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
I20260926 06:27:27.585427 15139 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:27.585515 15139 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
W20260926 06:27:27.585876 15139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:27:27.681259 15143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:27.681525 15143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:27.681582 15143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:27.683446 15143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:27.683518 15143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:27.683562 15143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:27:27.685497 15143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.39.212:40061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:46515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=46651
--tserver_master_addrs=127.2.39.254:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:27.685742 15143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:27.685992 15143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:27.692216 15149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:27.692378 15143 server_base.cc:1034] running on GCE node
W20260926 06:27:27.692863 15148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:27.692914 15151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:27.693301 15143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:27.693641 15143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:27.694859 15143 hybrid_clock.cc:648] HybridClock initialized: now 1790404047694832 us; error 45 us; skew 500 ppm
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:27.698871 15143 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:27:27.699393 15143 webserver.cc:460] Webserver started at http://127.2.39.194:46651/ using document root <none> and password file <none>
I20260926 06:27:27.699641 15143 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:27.699743 15143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:27.715273 15143 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.001s
I20260926 06:27:27.716284 15159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.716476 15143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:27:27.716570 15143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "d961e52de4e14707bdfe1066fd942a83"
format_stamp: "Formatted at 2026-09-26 06:27:26 on dist-test-slave-fkbn"
server_key: "cbcffe554019822f35a3fea3898d87e8"
server_key_iv: "e3116e6a99f2e7503b8e40156507a669"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:27.717007 15143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:27.740659 15143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:27.743837 15143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:27.743996 15143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:27.744220 15143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:27.744532 15143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:27.744585 15143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.744627 15143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:27.744653 15143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.750530 15143 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:46515
I20260926 06:27:27.750694 15272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:46515 every 8 connection(s)
I20260926 06:27:27.751003 15143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:27:27.751583 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 15143
I20260926 06:27:27.751884 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.39.212:40061/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:41971
--local_ip_for_outbound_sockets=127.2.39.195
--tserver_master_addrs=127.2.39.254:41391
--webserver_port=38729
--webserver_interface=127.2.39.195
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.39.212:40061/kms with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:27.764359 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:27.752288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:37463 (local address 127.2.39.254:41391)
0926 06:27:27.752335 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:27.752337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:27.753441 (+ 1104us) server_negotiation.cc:407] Connection header received
0926 06:27:27.754229 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:27.754230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:27.754247 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:27.754299 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:27.755050 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.755921 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.757755 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.758099 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.758148 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:27.760892 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:27.760894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:27.760895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:27.760903 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:27.762767 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:27.763292 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:27.763293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:27.763294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:27.763354 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:27.763674 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:27.763676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:27.763676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:27.763801 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:27.763842 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:27.764181 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 06:27:27.764262 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:27:27.765069 15275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:27.752011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41391 (local address 127.2.39.194:37463)
0926 06:27:27.753296 (+ 1285us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:27.753348 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:27.754129 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:27.754421 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:27.754425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:27.754669 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:27.754940 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:27.754946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.756189 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.756190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:27.757647 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:27.757650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.758264 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.758266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:27.758413 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:27.758982 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:27.758992 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:27.760774 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:27.762866 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:27.762872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:27.762875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:27.763192 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:27.763436 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:27.763437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:27.763437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:27.763583 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:27.763887 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:27.763893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:27.764040 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 06:27:27.764697 (+ 657us) client_negotiation.cc:240] Negotiation successful
0926 06:27:27.764848 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1239,"thread_start_us":76,"threads_started":1}
I20260926 06:27:27.765766 15273 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41391
I20260926 06:27:27.765895 15273 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:27.766105 15273 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
W20260926 06:27:27.766687 15273 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:41391 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 06:27:27.766793 15273 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:27.766880 15273 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
I20260926 06:27:27.767174 15273 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:27.767270 15273 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
W20260926 06:27:27.767546 15273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:27:27.853863 15276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:27.854067 15276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:27.854107 15276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:27.855329 15276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:27.855396 15276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:27.855440 15276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:27:27.857002 15276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.39.212:40061/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:41971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=38729
--tserver_master_addrs=127.2.39.254:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:27.857249 15276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:27.857506 15276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:27.862519 15282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:27.862499 15284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:27.862880 15281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:27.863351 15276 server_base.cc:1034] running on GCE node
I20260926 06:27:27.863547 15276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:27.863756 15276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:27.864914 15276 hybrid_clock.cc:648] HybridClock initialized: now 1790404047864890 us; error 37 us; skew 500 ppm
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:27.868163 15276 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:27:27.868652 15276 webserver.cc:460] Webserver started at http://127.2.39.195:38729/ using document root <none> and password file <none>
I20260926 06:27:27.868903 15276 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:27.868991 15276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:27.895799 15276 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 06:27:27.896842 15293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.897045 15276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:27.897135 15276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "7b1f5a243d0e4f59993fae7ae46d1100"
format_stamp: "Formatted at 2026-09-26 06:27:27 on dist-test-slave-fkbn"
server_key: "8b0a89a118fceb57e07e3ae0c03e94d2"
server_key_iv: "a3aebc0145a26211ef014a74c6fdaba8"
server_key_version: "kuduclusterkey@0"
I20260926 06:27:27.897658 15276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:27.930089 15276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:27.932029 15276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:27.932168 15276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:27.932417 15276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:27.932744 15276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:27.932787 15276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.932816 15276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:27.932881 15276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:27.938874 15276 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:41971
I20260926 06:27:27.939029 15406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:41971 every 8 connection(s)
I20260926 06:27:27.939368 15276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:27:27 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404047, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:27.948670 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 15276
I20260926 06:27:27.952721 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:27.942211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41391 (local address 127.2.39.195:54291)
0926 06:27:27.942446 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:27.942476 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:27.943094 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:27.943445 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:27.943449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:27.943695 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:27.943890 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:27.943895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.944774 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.944776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:27.945886 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:27.945889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.946394 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.946395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:27.946529 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:27.947063 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:27.947071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:27.949055 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:27.950998 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:27.951004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:27.951007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:27.951297 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:27.951568 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:27.951569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:27.951569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:27.951718 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:27.952035 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:27.952041 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:27.952198 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 06:27:27.952313 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 06:27:27.952490 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":95,"threads_started":1}
I20260926 06:27:27.953047 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:27.942223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:54291 (local address 127.2.39.254:41391)
0926 06:27:27.942305 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:27.942308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:27.942567 (+ 259us) server_negotiation.cc:407] Connection header received
0926 06:27:27.943224 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:27.943226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:27.943238 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:27.943268 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:27.943994 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.944664 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.945984 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.946285 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.946386 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:27.949169 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:27.949170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:27.949171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:27.949178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:27.950905 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:27.951400 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:27.951401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:27.951402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:27.951472 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:27.951801 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:27.951802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:27.951803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:27.951943 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:27.952007 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:27.952874 (+ 867us) server_negotiation.cc:299] Negotiation successful
0926 06:27:27.952958 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 06:27:27.953519 15407 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41391
I20260926 06:27:27.953683 15407 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:27.953934 15407 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
W20260926 06:27:27.954610 15407 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:41391 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 06:27:27.954735 15407 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:27.954819 15407 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
I20260926 06:27:27.955231 15407 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:27.955322 15407 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
W20260926 06:27:27.955825 15407 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:27:27.959728 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:27.950108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.2.39.254:41391)
0926 06:27:27.950235 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:27.950238 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:27.951559 (+ 1321us) server_negotiation.cc:407] Connection header received
0926 06:27:27.951582 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:27.951583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:27.951594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:27.951651 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:27.952402 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.953552 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.954680 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:27.955139 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:27.955197 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:27.956941 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:27.956942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:27.956943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:27.956949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:27.958646 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:27.958962 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:27.958963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:27.958964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:27.959024 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:27.959206 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:27.959207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:27.959207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:27.959337 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:27.959515 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:27.959557 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 06:27:27.959638 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
Sep 26 06:27:28 dist-test-slave-fkbn krb5kdc[13945](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790404048, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:28.586076 15139 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:28.586249 15139 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
I20260926 06:27:28.586674 14941 ts_manager.cc:194] Registered new tserver with Master: 85b512f6af774e2799456bc7b6b2a457 (127.2.39.193:45675)
I20260926 06:27:28.587721 14941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:42109
I20260926 06:27:28.767750 15273 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:28.767907 15273 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
I20260926 06:27:28.768186 14941 ts_manager.cc:194] Registered new tserver with Master: d961e52de4e14707bdfe1066fd942a83 (127.2.39.194:46515)
I20260926 06:27:28.768714 14941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:37463
Sep 26 06:27:28 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404048, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:28.956058 15407 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:28.956212 15407 heartbeater.cc:511] Master 127.2.39.254:41391 requested a full tablet report, sending...
I20260926 06:27:28.956537 14941 ts_manager.cc:194] Registered new tserver with Master: 7b1f5a243d0e4f59993fae7ae46d1100 (127.2.39.195:41971)
I20260926 06:27:28.956946 14941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:54291
I20260926 06:27:28.966315 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:27:28.976166 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:28.967472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33704 (local address 127.2.39.254:41391)
0926 06:27:28.967676 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:28.967679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:28.967781 (+ 102us) server_negotiation.cc:407] Connection header received
0926 06:27:28.967878 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:28.967879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:28.967930 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:28.967976 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:28.968726 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:28.969549 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:28.970636 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:28.970964 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:28.971006 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:28.972606 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:28.972607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:28.972608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:28.972615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:28.974425 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:28.974849 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:28.974850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:28.974851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:28.974926 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:28.975226 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:28.975231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:28.975231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:28.975362 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:28.975445 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:28.975924 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 06:27:28.976015 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":66,"threads_started":1}
I20260926 06:27:28.978583 14941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:27:28.981366 14941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:27:28.995457 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:28.988254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:46515 (local address 127.0.0.1:40776)
0926 06:27:28.988993 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:28.989001 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:28.989047 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:28.989781 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:28.989782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:28.989784 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:28.989821 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:28.989823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:28.992515 (+ 2692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:28.992516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:28.994117 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:28.994120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:28.995151 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:28.995152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:28.995244 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:28.995257 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 06:27:28.995324 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:27:28.995372 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":66,"threads_started":1}
I20260926 06:27:28.998451 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:28.988338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.2.39.194:46515)
0926 06:27:28.989547 (+ 1209us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:28.989552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:28.989561 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:27:28.989610 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:28.989613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:28.989627 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:28.989696 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:28.990758 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:28.991799 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:28.994217 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:28.994955 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:28.995040 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:28.995116 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:28.998319 (+ 3203us) server_negotiation.cc:299] Negotiation successful
0926 06:27:28.998370 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":53,"threads_started":1}
I20260926 06:27:28.999245 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:28.988029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:41971 (local address 127.0.0.1:33554)
0926 06:27:28.988592 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:28.988634 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:28.988885 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:28.989361 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:28.989364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:28.989366 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:28.989430 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:28.989431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:28.996231 (+ 6800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:28.996232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:28.997295 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:28.997298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:28.998743 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:28.998744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:28.998811 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:28.999095 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 06:27:28.999127 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 06:27:28.999170 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":64,"threads_started":1}
I20260926 06:27:28.999794 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:28.988111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.2.39.195:41971)
0926 06:27:28.989144 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:28.989149 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:28.989158 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:27:28.989192 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:28.989194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:28.989208 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:28.989275 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:28.989896 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:28.992420 (+ 2524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:28.997403 (+ 4983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:28.998221 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:28.999573 (+ 1352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:28.999646 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:28.999673 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 06:27:28.999719 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":54,"threads_started":1}
I20260926 06:27:29.000705 15201 tablet_service.cc:1467] Processing CreateTablet for tablet ca6f5c49d4854a20946fb34ae0d14f4c (DEFAULT_TABLE table=test-table [id=94190ea9067841328507c78b3fd3d847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:29.001082 15201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6f5c49d4854a20946fb34ae0d14f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:29.004261 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:28.988419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45675 (local address 127.0.0.1:37814)
0926 06:27:28.991496 (+ 3077us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:28.991507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:28.991566 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:29.000506 (+ 8940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:29.000507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:29.000510 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:29.000555 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:29.000556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.001771 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.001772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:29.002796 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:29.002799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.004055 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.004056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:29.004119 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:29.004124 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:27:29.004153 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 06:27:29.004191 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3057,"thread_start_us":76,"threads_started":1}
I20260926 06:27:29.004622 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:28.988532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.2.39.193:45675)
0926 06:27:29.000326 (+ 11794us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:29.000332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:29.000343 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:27:29.000381 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:29.000383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:29.000398 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:29.000658 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:29.000739 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.001676 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.002906 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.003938 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.004408 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:29.004479 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:29.004506 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 06:27:29.004545 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11757,"thread_start_us":49,"threads_started":1}
I20260926 06:27:29.004829 15440 tablet_bootstrap.cc:492] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: Bootstrap starting.
I20260926 06:27:29.005254 15055 tablet_service.cc:1467] Processing CreateTablet for tablet ca6f5c49d4854a20946fb34ae0d14f4c (DEFAULT_TABLE table=test-table [id=94190ea9067841328507c78b3fd3d847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:29.005470 15440 tablet_bootstrap.cc:654] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:29.005622 15055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6f5c49d4854a20946fb34ae0d14f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:29.005625 15341 tablet_service.cc:1467] Processing CreateTablet for tablet ca6f5c49d4854a20946fb34ae0d14f4c (DEFAULT_TABLE table=test-table [id=94190ea9067841328507c78b3fd3d847]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:29.005795 15440 log.cc:824] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:29.005944 15341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6f5c49d4854a20946fb34ae0d14f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:29.007633 15440 tablet_bootstrap.cc:492] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: No bootstrap required, opened a new log
I20260926 06:27:29.007704 15440 ts_tablet_manager.cc:1397] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 06:27:29.010124 15440 raft_consensus.cc:348] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 41971 } } peers { permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 } } peers { permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 } }
I20260926 06:27:29.010243 15440 raft_consensus.cc:374] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:29.010263 15440 raft_consensus.cc:729] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d961e52de4e14707bdfe1066fd942a83, State: Initialized, Role: FOLLOWER
I20260926 06:27:29.010383 15440 consensus_queue.cc:260] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 41971 } } peers { permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 } } peers { permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 } }
I20260926 06:27:29.010586 15440 ts_tablet_manager.cc:1428] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 06:27:29.010710 15273 heartbeater.cc:503] Master 127.2.39.254:41391 was elected leader, sending a full tablet report...
I20260926 06:27:29.011281 15443 tablet_bootstrap.cc:492] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: Bootstrap starting.
I20260926 06:27:29.011871 15443 tablet_bootstrap.cc:654] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:29.012164 15443 log.cc:824] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:29.012560 15442 tablet_bootstrap.cc:492] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457: Bootstrap starting.
I20260926 06:27:29.013230 15442 tablet_bootstrap.cc:654] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:29.013410 15443 tablet_bootstrap.cc:492] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: No bootstrap required, opened a new log
I20260926 06:27:29.013530 15443 ts_tablet_manager.cc:1397] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:27:29.013589 15442 log.cc:824] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:29.014439 15442 tablet_bootstrap.cc:492] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457: No bootstrap required, opened a new log
I20260926 06:27:29.014509 15442 ts_tablet_manager.cc:1397] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:27:29.016746 15443 raft_consensus.cc:348] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 41971 } } peers { permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 } } peers { permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 } }
I20260926 06:27:29.016901 15443 raft_consensus.cc:374] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:29.016932 15443 raft_consensus.cc:729] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b1f5a243d0e4f59993fae7ae46d1100, State: Initialized, Role: FOLLOWER
I20260926 06:27:29.017079 15443 consensus_queue.cc:260] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 41971 } } peers { permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 } } peers { permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 } }
I20260926 06:27:29.017606 15443 ts_tablet_manager.cc:1428] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 06:27:29.017766 15407 heartbeater.cc:503] Master 127.2.39.254:41391 was elected leader, sending a full tablet report...
I20260926 06:27:29.017710 15442 raft_consensus.cc:348] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 41971 } } peers { permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 } } peers { permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 } }
I20260926 06:27:29.017860 15442 raft_consensus.cc:374] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:29.017894 15442 raft_consensus.cc:729] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b512f6af774e2799456bc7b6b2a457, State: Initialized, Role: FOLLOWER
I20260926 06:27:29.018002 15442 consensus_queue.cc:260] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 41971 } } peers { permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 } } peers { permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 } }
I20260926 06:27:29.018262 15442 ts_tablet_manager.cc:1428] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 06:27:29.018365 15139 heartbeater.cc:503] Master 127.2.39.254:41391 was elected leader, sending a full tablet report...
W20260926 06:27:29.066799 15140 tablet.cc:2369] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:27:29.115248 15447 raft_consensus.cc:482] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:27:29.115396 15447 raft_consensus.cc:504] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 41971 } } peers { permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 } } peers { permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 } }
I20260926 06:27:29.115777 15447 leader_election.cc:288] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85b512f6af774e2799456bc7b6b2a457 (127.2.39.193:45675), d961e52de4e14707bdfe1066fd942a83 (127.2.39.194:46515)
Sep 26 06:27:29 dist-test-slave-fkbn krb5kdc[13945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404048, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for HTTP/127.2.39.212@KRBTEST.COM
I20260926 06:27:29.121938 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:29.116335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:34695 (local address 127.2.39.194:46515)
0926 06:27:29.116393 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:29.116395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:29.116557 (+ 162us) server_negotiation.cc:407] Connection header received
0926 06:27:29.116656 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:29.116657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:29.116668 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:29.116695 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:29.116900 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.117638 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.119103 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.119640 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.119714 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:29.119724 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:29.121820 (+ 2096us) server_negotiation.cc:299] Negotiation successful
0926 06:27:29.121867 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:27:29.122169 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:29.116135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:46515 (local address 127.2.39.195:34695)
0926 06:27:29.116500 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:29.116509 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:29.116584 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:29.116758 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:29.116758 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:29.116760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:29.116831 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:29.116833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.117708 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.117709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:29.119005 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:29.119009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.121646 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.121647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:29.121720 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:29.121725 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:27:29.122028 (+ 303us) client_negotiation.cc:240] Negotiation successful
0926 06:27:29.122076 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":81,"threads_started":1}
I20260926 06:27:29.122682 15211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6f5c49d4854a20946fb34ae0d14f4c" candidate_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961e52de4e14707bdfe1066fd942a83" is_pre_election: true
I20260926 06:27:29.122874 15211 raft_consensus.cc:2454] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b1f5a243d0e4f59993fae7ae46d1100 in term 0.
I20260926 06:27:29.123126 15296 leader_election.cc:302] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1f5a243d0e4f59993fae7ae46d1100, d961e52de4e14707bdfe1066fd942a83; no voters:
I20260926 06:27:29.123266 15447 raft_consensus.cc:2781] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:27:29.123317 15447 raft_consensus.cc:482] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:27:29.123339 15447 raft_consensus.cc:3037] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:29.124287 15447 raft_consensus.cc:504] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 41971 } } peers { permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 } } peers { permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 } }
I20260926 06:27:29.124425 15447 leader_election.cc:288] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [CANDIDATE]: Term 1 election: Requested vote from peers 85b512f6af774e2799456bc7b6b2a457 (127.2.39.193:45675), d961e52de4e14707bdfe1066fd942a83 (127.2.39.194:46515)
I20260926 06:27:29.124675 15211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6f5c49d4854a20946fb34ae0d14f4c" candidate_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d961e52de4e14707bdfe1066fd942a83"
I20260926 06:27:29.124780 15211 raft_consensus.cc:3037] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:29.125440 15211 raft_consensus.cc:2454] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b1f5a243d0e4f59993fae7ae46d1100 in term 1.
I20260926 06:27:29.125635 15296 leader_election.cc:302] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b1f5a243d0e4f59993fae7ae46d1100, d961e52de4e14707bdfe1066fd942a83; no voters:
I20260926 06:27:29.125761 15447 raft_consensus.cc:2781] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:27:29.125907 15447 raft_consensus.cc:686] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 1 LEADER]: Becoming Leader. State: Replica: 7b1f5a243d0e4f59993fae7ae46d1100, State: Running, Role: LEADER
I20260926 06:27:29.125993 15447 consensus_queue.cc:237] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 41971 } } peers { permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 } } peers { permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 } }
I20260926 06:27:29.126775 14940 catalog_manager.cc:5495] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b1f5a243d0e4f59993fae7ae46d1100 (127.2.39.195). New cstate: current_term: 1 leader_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 41971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 } health_report { overall_health: UNKNOWN } } }
I20260926 06:27:29.128991 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:29.116509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:35811 (local address 127.2.39.193:45675)
0926 06:27:29.116562 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:29.116564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:29.122398 (+ 5834us) server_negotiation.cc:407] Connection header received
0926 06:27:29.122432 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:29.122433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:29.122446 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:29.122505 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:29.123300 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.124001 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.128028 (+ 4027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.128600 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.128673 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:29.128685 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:29.128858 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 06:27:29.128908 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:27:29.129354 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:29.116389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45675 (local address 127.2.39.195:35811)
0926 06:27:29.122305 (+ 5916us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:29.122314 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:29.122361 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:29.123168 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:29.123169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:29.123171 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:29.123202 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:29.123204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.124108 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.124109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:29.127922 (+ 3813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:29.127925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.128715 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.128716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:29.128772 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:29.128775 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:27:29.129256 (+ 481us) client_negotiation.cc:240] Negotiation successful
0926 06:27:29.129301 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5904,"thread_start_us":55,"threads_started":1}
I20260926 06:27:29.129693 15089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6f5c49d4854a20946fb34ae0d14f4c" candidate_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b512f6af774e2799456bc7b6b2a457"
I20260926 06:27:29.129714 15090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6f5c49d4854a20946fb34ae0d14f4c" candidate_uuid: "7b1f5a243d0e4f59993fae7ae46d1100" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b512f6af774e2799456bc7b6b2a457" is_pre_election: true
I20260926 06:27:29.129817 15089 raft_consensus.cc:3037] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:29.130967 15089 raft_consensus.cc:2454] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b1f5a243d0e4f59993fae7ae46d1100 in term 1.
I20260926 06:27:29.138715 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:29.135780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.2.39.195:41971)
0926 06:27:29.135819 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:29.135820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:29.135828 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:27:29.135861 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:29.135861 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:29.135881 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:29.135952 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:27:29.136086 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.136717 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.137790 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:29.138054 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:29.138118 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:29.138380 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:27:29.138478 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:27:29.138593 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:29.138621 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 06:27:29.138642 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:27:29.140578 15211 raft_consensus.cc:1261] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83 [term 1 FOLLOWER]: Refusing update from remote peer 7b1f5a243d0e4f59993fae7ae46d1100: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:27:29.140973 15452 consensus_queue.cc:1035] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d961e52de4e14707bdfe1066fd942a83" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 46515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:27:29.143266 15089 raft_consensus.cc:1261] T ca6f5c49d4854a20946fb34ae0d14f4c P 85b512f6af774e2799456bc7b6b2a457 [term 1 FOLLOWER]: Refusing update from remote peer 7b1f5a243d0e4f59993fae7ae46d1100: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:27:29.143591 15452 consensus_queue.cc:1035] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b512f6af774e2799456bc7b6b2a457" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:27:29.147322 15455 mvcc.cc:205] Tried to move back new op lower bound from 7333494985276870656 to 7333494985220538368. Current Snapshot: MvccSnapshot[applied={T|T < 7333494985276870656}]
I20260926 06:27:29.147888 15458 mvcc.cc:205] Tried to move back new op lower bound from 7333494985276870656 to 7333494985220538368. Current Snapshot: MvccSnapshot[applied={T|T < 7333494985276870656}]
I20260926 06:27:29.149785 15456 mvcc.cc:205] Tried to move back new op lower bound from 7333494985276870656 to 7333494985220538368. Current Snapshot: MvccSnapshot[applied={T|T < 7333494985276870656}]
I20260926 06:27:29.150995 14940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:27:29.151091 14940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33704:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:27:29.151930 14940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7beeaf8688cf404285ebfaf198cf1d49: Sending DeleteTablet for 3 replicas of tablet ca6f5c49d4854a20946fb34ae0d14f4c
I20260926 06:27:29.152346 15341 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6f5c49d4854a20946fb34ae0d14f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:29 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:33554
I20260926 06:27:29.152557 15201 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6f5c49d4854a20946fb34ae0d14f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:29 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:40776
I20260926 06:27:29.152927 15055 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6f5c49d4854a20946fb34ae0d14f4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:29 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:37814
I20260926 06:27:29.153556 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 14979
I20260926 06:27:29.157612 15464 tablet_replica.cc:330] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: stopping tablet replica
I20260926 06:27:29.157716 15464 raft_consensus.cc:2229] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:27:29.157830 15464 raft_consensus.cc:2258] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:29.157979 15465 tablet_replica.cc:330] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: stopping tablet replica
I20260926 06:27:29.158088 15465 raft_consensus.cc:2229] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:27:29.158154 15465 raft_consensus.cc:2258] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:29.161967 15464 ts_tablet_manager.cc:1905] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:27:29.162868 15465 ts_tablet_manager.cc:1905] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:27:29.164458 15465 ts_tablet_manager.cc:1918] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:27:29.164599 15465 log.cc:1196] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/ca6f5c49d4854a20946fb34ae0d14f4c
I20260926 06:27:29.164923 15465 ts_tablet_manager.cc:1939] T ca6f5c49d4854a20946fb34ae0d14f4c P d961e52de4e14707bdfe1066fd942a83: Deleting consensus metadata
I20260926 06:27:29.165450 14927 catalog_manager.cc:4841] TS d961e52de4e14707bdfe1066fd942a83 (127.2.39.194:46515): tablet ca6f5c49d4854a20946fb34ae0d14f4c (table test-table [id=94190ea9067841328507c78b3fd3d847]) successfully deleted
I20260926 06:27:29.165503 15464 ts_tablet_manager.cc:1918] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:27:29.165593 15464 log.cc:1196] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/ca6f5c49d4854a20946fb34ae0d14f4c
I20260926 06:27:29.165838 15464 ts_tablet_manager.cc:1939] T ca6f5c49d4854a20946fb34ae0d14f4c P 7b1f5a243d0e4f59993fae7ae46d1100: Deleting consensus metadata
I20260926 06:27:29.166239 14926 catalog_manager.cc:4841] TS 7b1f5a243d0e4f59993fae7ae46d1100 (127.2.39.195:41971): tablet ca6f5c49d4854a20946fb34ae0d14f4c (table test-table [id=94190ea9067841328507c78b3fd3d847]) successfully deleted
W20260926 06:27:29.166280 14925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:45675 (error 108)
I20260926 06:27:29.166756 15438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:29.166620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45675 (local address 127.0.0.1:37830)
0926 06:27:29.166681 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:29.166718 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:45675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 06:27:29.166858 14925 catalog_manager.cc:4584] TS 85b512f6af774e2799456bc7b6b2a457 (127.2.39.193:45675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca6f5c49d4854a20946fb34ae0d14f4c: Network error: Client connection negotiation failed: client connection to 127.2.39.193:45675: connect: Connection refused (error 111)
I20260926 06:27:29.172225 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 15143
I20260926 06:27:29.179018 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 15276
I20260926 06:27:29.186194 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 14908
I20260926 06:27:29.197422 2207 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 06:27:29.542501 2207 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 06:27:29.542645 2207 mini_ranger.cc:66] Stopping Ranger...
I20260926 06:27:29.886866 2207 mini_ranger.cc:68] Stopped Ranger
2026-09-26 06:27:29.887 UTC [13975] LOG: received smart shutdown request
2026-09-26 06:27:29.891 UTC [13975] LOG: background worker "logical replication launcher" (PID 13984) exited with exit code 1
2026-09-26 06:27:29.891 UTC [13979] LOG: shutting down
2026-09-26 06:27:30.086 UTC [13975] LOG: database system is shut down
2026-09-26T06:27:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48170 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:27:30 dist-test-slave-fkbn krb5kdc[15477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:27:30 dist-test-slave-fkbn krb5kdc[15477](info): set up 2 sockets
Sep 26 06:27:30 dist-test-slave-fkbn krb5kdc[15477](info): commencing operation
krb5kdc: starting...
W20260926 06:27:31.233520 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:27:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:27:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:27:31.350484 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:39749
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:39749,127.2.39.253:43279,127.2.39.252:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:31.429381 15490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:31.429592 15490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:31.429659 15490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:31.430768 15490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:27:31.430802 15490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:31.430816 15490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:31.430828 15490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:27:31.430840 15490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:27:31.432225 15490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:39749,127.2.39.253:43279,127.2.39.252:36971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:39749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:31.432410 15490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:31.432569 15490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:31.437292 15498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:31.437304 15496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:31.437414 15490 server_base.cc:1034] running on GCE node
W20260926 06:27:31.437361 15495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:31.437814 15490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:31.438100 15490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:31.439260 15490 hybrid_clock.cc:648] HybridClock initialized: now 1790404051439254 us; error 49 us; skew 500 ppm
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:31.445273 15490 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:27:31.445797 15490 webserver.cc:460] Webserver started at http://127.2.39.254:39671/ using document root <none> and password file <none>
I20260926 06:27:31.446034 15490 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:31.446102 15490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:31.446259 15490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:31.447599 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "99cde807dceb48aa8b603c977efce5fc"
format_stamp: "Formatted at 2026-09-26 06:27:31 on dist-test-slave-fkbn"
server_key: "44f9ebb621d6303f4c0ec93e16540ae1"
server_key_iv: "473c2828708992b293774ee2027ebbd3"
server_key_version: "encryptionkey@0"
I20260926 06:27:31.447994 15490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "99cde807dceb48aa8b603c977efce5fc"
format_stamp: "Formatted at 2026-09-26 06:27:31 on dist-test-slave-fkbn"
server_key: "44f9ebb621d6303f4c0ec93e16540ae1"
server_key_iv: "473c2828708992b293774ee2027ebbd3"
server_key_version: "encryptionkey@0"
I20260926 06:27:31.449281 15490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 06:27:31.450076 15505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:31.450248 15490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:31.450341 15490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "99cde807dceb48aa8b603c977efce5fc"
format_stamp: "Formatted at 2026-09-26 06:27:31 on dist-test-slave-fkbn"
server_key: "44f9ebb621d6303f4c0ec93e16540ae1"
server_key_iv: "473c2828708992b293774ee2027ebbd3"
server_key_version: "encryptionkey@0"
I20260926 06:27:31.450425 15490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:31.458475 15490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:31.460170 15490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:31.460330 15490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:31.464195 15490 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:39749
I20260926 06:27:31.464222 15557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:39749 every 8 connection(s)
I20260926 06:27:31.464550 15490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:27:31.465345 15558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:31.466884 15558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:31.474628 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 15490
I20260926 06:27:31.474715 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:27:31.474896 2207 external_mini_cluster.cc:1428] Setting key 6ed3c19c0bfc1a156624e3143c7e20cb
I20260926 06:27:31.476243 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.467663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.2.39.254:39749)
0926 06:27:31.467767 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.467772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.467809 (+ 37us) server_negotiation.cc:407] Connection header received
0926 06:27:31.468357 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.468359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.468375 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.468466 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.469027 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.469863 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.470742 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.471060 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.471184 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.473003 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.473009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.473010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.473016 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.474387 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.474898 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.474900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.474901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.475022 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.475288 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.475289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.475290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.475548 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.475651 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.475855 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.476000 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":38,"threads_started":1}
I20260926 06:27:31.476243 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.467526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:39749 (local address 127.0.0.1:60302)
0926 06:27:31.467667 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.467719 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:31.468187 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:31.468478 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:31.468480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:31.468770 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:31.468922 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.468926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.469953 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.469954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.470625 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.470627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.471159 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.471160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.471225 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.471672 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:31.471680 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:31.472924 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:31.474485 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.474492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.474493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.474758 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.475123 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.475124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.475124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.475216 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.475630 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:31.475635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:31.475697 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 06:27:31.475816 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:27:31.476065 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":102,"thread_start_us":61,"threads_started":1}
I20260926 06:27:31.476974 15558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:27:31.477339 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.477208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:43279 (local address 127.0.0.1:51680)
0926 06:27:31.477257 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.477303 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:27:31.477473 15507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.39.253:43279: connect: Connection refused (error 111)
I20260926 06:27:31.477952 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.477849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:43279 (local address 127.0.0.1:51692)
0926 06:27:31.477902 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.477929 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 06:27:31.478107 15558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.253:43279: Network error: Client connection negotiation failed: client connection to 127.2.39.253:43279: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.39.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab.
Entry for principal kudu/127.2.39.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab.
I20260926 06:27:31.498404 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.253:43279
--webserver_interface=127.2.39.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab
--principal=kudu/127.2.39.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:39749,127.2.39.253:43279,127.2.39.252:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:27:31.527308 15558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } attempt: 1
I20260926 06:27:31.527789 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.527654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:43279 (local address 127.0.0.1:51700)
0926 06:27:31.527721 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.527759 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 06:27:31.528170 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.528073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:43279 (local address 127.0.0.1:51708)
0926 06:27:31.528121 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.528147 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 06:27:31.528255 15558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.253:43279: Network error: Client connection negotiation failed: client connection to 127.2.39.253:43279: connect: Connection refused (error 111)
W20260926 06:27:31.578476 15565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:31.578678 15565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:31.578701 15565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:31.579836 15565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:27:31.579876 15565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:31.579890 15565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:31.579939 15565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:27:31.579958 15565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:27:31.581426 15565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:39749,127.2.39.253:43279,127.2.39.252:36971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab
--principal=kudu/127.2.39.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.253:43279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.39.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:31.581692 15565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:31.581911 15565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:31.586740 15570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:31.586750 15573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:31.586833 15571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:31.586871 15565 server_base.cc:1034] running on GCE node
I20260926 06:27:31.587110 15565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:31.587416 15565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:31.588599 15565 hybrid_clock.cc:648] HybridClock initialized: now 1790404051588579 us; error 37 us; skew 500 ppm
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:31.591089 15565 init.cc:377] Logged in from keytab as kudu/127.2.39.253@KRBTEST.COM (short username kudu)
I20260926 06:27:31.591552 15565 webserver.cc:460] Webserver started at http://127.2.39.253:39123/ using document root <none> and password file <none>
I20260926 06:27:31.591847 15565 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:31.591908 15565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:31.592051 15565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:31.593073 15565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/data/instance:
uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2"
format_stamp: "Formatted at 2026-09-26 06:27:31 on dist-test-slave-fkbn"
server_key: "315d358f740a35fb484dc263861068b9"
server_key_iv: "715053d2dc538db15e2edf0adf542b45"
server_key_version: "encryptionkey@0"
I20260926 06:27:31.593461 15565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/wal/instance:
uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2"
format_stamp: "Formatted at 2026-09-26 06:27:31 on dist-test-slave-fkbn"
server_key: "315d358f740a35fb484dc263861068b9"
server_key_iv: "715053d2dc538db15e2edf0adf542b45"
server_key_version: "encryptionkey@0"
I20260926 06:27:31.594815 15565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:27:31.595561 15580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:31.595799 15565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:27:31.595883 15565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/wal
uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2"
format_stamp: "Formatted at 2026-09-26 06:27:31 on dist-test-slave-fkbn"
server_key: "315d358f740a35fb484dc263861068b9"
server_key_iv: "715053d2dc538db15e2edf0adf542b45"
server_key_version: "encryptionkey@0"
I20260926 06:27:31.595952 15565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:31.601629 15558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } attempt: 2
I20260926 06:27:31.602042 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.601911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:43279 (local address 127.0.0.1:51724)
0926 06:27:31.601956 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.601986 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 06:27:31.602425 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.602333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:43279 (local address 127.0.0.1:51740)
0926 06:27:31.602378 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.602403 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:43279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:27:31.602535 15558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.253:43279: Network error: Client connection negotiation failed: client connection to 127.2.39.253:43279: connect: Connection refused (error 111)
I20260926 06:27:31.604712 15565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:31.607510 15565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:31.607702 15565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:31.612874 15565 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.253:43279
I20260926 06:27:31.612937 15632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.253:43279 every 8 connection(s)
I20260926 06:27:31.613374 15565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
I20260926 06:27:31.614229 15633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:31.615816 15633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:27:31.622526 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 15565
I20260926 06:27:31.622633 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-1/wal/instance
I20260926 06:27:31.622788 2207 external_mini_cluster.cc:1428] Setting key 1b771fa55e201fd16267e849ac3a4293
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:31.627398 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.616454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60308 (local address 127.2.39.254:39749)
0926 06:27:31.616509 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.616512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.616767 (+ 255us) server_negotiation.cc:407] Connection header received
0926 06:27:31.617466 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.617467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.617490 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.617565 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.618611 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.619088 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.620757 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.621074 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.621116 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.623669 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.623671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.623671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.623678 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.625830 (+ 2152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.626320 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.626322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.626322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.626386 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.626678 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.626680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.626680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.626812 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.626881 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.627231 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.627313 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 06:27:31.627606 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.616447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:39749 (local address 127.0.0.1:60308)
0926 06:27:31.616629 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.616680 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:31.617339 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:31.617695 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:31.617699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:31.618134 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:31.618455 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.618460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.619193 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.619194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.620648 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.620651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.621200 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.621201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.621332 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.621879 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:31.621888 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:31.623552 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:31.625918 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.625925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.625927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.626215 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.626465 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.626466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.626466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.626607 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.626895 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:31.626902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:31.627076 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 06:27:31.627195 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 06:27:31.627357 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260926 06:27:31.628172 15633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.39.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.252@KRBTEST.COM" created.
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
I20260926 06:27:31.639220 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.628437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:43279 (local address 127.0.0.1:51756)
0926 06:27:31.628496 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.628508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:31.628551 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:31.628820 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:31.628821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:31.628993 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:31.629059 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.629061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.630112 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.630113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.630927 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.630930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.631585 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.631586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.631673 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.632134 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:31.632139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:31.636057 (+ 3918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:31.638086 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.638087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.638088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.638241 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.638484 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.638485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.638486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.638552 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.638930 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:31.638931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:31.638990 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 06:27:31.639067 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 06:27:31.639149 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 06:27:31.639360 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.628495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51756 (local address 127.2.39.253:43279)
0926 06:27:31.628669 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.628674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.628692 (+ 18us) server_negotiation.cc:407] Connection header received
0926 06:27:31.628719 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.628720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.628732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.628792 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.629134 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.630014 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.631035 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.631499 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.631580 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.636138 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.636146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.636146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.636153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.638013 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.638320 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.638323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.638323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.638386 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.638628 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.638629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.638629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.638815 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.638905 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.639129 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.639216 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260926 06:27:31.640249 15633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:27:31.640630 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.640509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50122)
0926 06:27:31.640558 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.640610 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:27:31.640799 15584 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
I20260926 06:27:31.641218 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.641129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50132)
0926 06:27:31.641175 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.641202 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 06:27:31.641350 15633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:36971: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab.
Entry for principal kudu/127.2.39.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab.
I20260926 06:27:31.651907 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.252:36971
--webserver_interface=127.2.39.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab
--principal=kudu/127.2.39.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:39749,127.2.39.253:43279,127.2.39.252:36971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:27:31.690611 15633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } attempt: 1
I20260926 06:27:31.691120 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.690965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50134)
0926 06:27:31.691040 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.691085 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 06:27:31.691537 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.691392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50150)
0926 06:27:31.691470 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.691504 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 06:27:31.691648 15633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:36971: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
I20260926 06:27:31.693666 15558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } attempt: 3
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
I20260926 06:27:31.701624 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.694005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.2.39.253:43279)
0926 06:27:31.694080 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.694083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.694120 (+ 37us) server_negotiation.cc:407] Connection header received
0926 06:27:31.694188 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.694189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.694201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.694298 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.694678 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.695552 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.696206 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.696518 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.696590 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.698656 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.698658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.698658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.698665 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.700431 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.700810 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.700811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.700811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.700871 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.701075 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.701076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.701076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.701208 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.701274 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.701439 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.701517 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:27:31.701643 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.693965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:43279 (local address 127.0.0.1:51760)
0926 06:27:31.694072 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.694081 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:31.694130 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:31.694313 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:31.694314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:31.694510 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:31.694604 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.694607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.695630 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.695631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.696123 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.696124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.696592 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.696593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.696650 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.697115 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:31.697119 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:31.698571 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:31.700507 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.700509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.700509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.700729 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.700928 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.700929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.700929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.701003 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.701266 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:31.701267 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:31.701332 (+ 65us) client_negotiation.cc:769] Sending connection context
0926 06:27:31.701406 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:27:31.701503 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 06:27:31.702400 15558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:27:31.702755 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.702647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50158)
0926 06:27:31.702690 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.702724 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 06:27:31.703063 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.702975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50172)
0926 06:27:31.703019 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.703042 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 06:27:31.703174 15558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:36971: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
I20260926 06:27:31.734331 15558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } attempt: 1
I20260926 06:27:31.734794 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.734670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50184)
0926 06:27:31.734734 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.734768 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 06:27:31.735142 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.735032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50196)
0926 06:27:31.735095 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.735125 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 06:27:31.735234 15558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:36971: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
W20260926 06:27:31.736410 15640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:31.736631 15640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:31.736677 15640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:31.737828 15640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:27:31.737884 15640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:31.737911 15640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:31.737943 15640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:27:31.737965 15640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:27:31.739396 15640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:39749,127.2.39.253:43279,127.2.39.252:36971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab
--principal=kudu/127.2.39.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.252:36971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.39.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:31.739668 15640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:31.739886 15640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:31.744458 15646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:31.744531 15645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:31.744445 15648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:31.744810 15640 server_base.cc:1034] running on GCE node
I20260926 06:27:31.745097 15640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:31.745421 15640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:31.746590 15640 hybrid_clock.cc:648] HybridClock initialized: now 1790404051746568 us; error 31 us; skew 500 ppm
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:31.749752 15640 init.cc:377] Logged in from keytab as kudu/127.2.39.252@KRBTEST.COM (short username kudu)
I20260926 06:27:31.750358 15640 webserver.cc:460] Webserver started at http://127.2.39.252:36773/ using document root <none> and password file <none>
I20260926 06:27:31.750612 15640 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:31.750680 15640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:31.750811 15640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:31.752022 15640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/data/instance:
uuid: "9b87c2b5a3a341998b50fb0c72268505"
format_stamp: "Formatted at 2026-09-26 06:27:31 on dist-test-slave-fkbn"
server_key: "de00aebf9f10e28c919b7cbf00348521"
server_key_iv: "f4603ab8fb374c87285993a6e922d3ba"
server_key_version: "encryptionkey@0"
I20260926 06:27:31.752444 15640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/wal/instance:
uuid: "9b87c2b5a3a341998b50fb0c72268505"
format_stamp: "Formatted at 2026-09-26 06:27:31 on dist-test-slave-fkbn"
server_key: "de00aebf9f10e28c919b7cbf00348521"
server_key_iv: "f4603ab8fb374c87285993a6e922d3ba"
server_key_version: "encryptionkey@0"
I20260926 06:27:31.754065 15640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:27:31.754837 15655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:31.755033 15640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:31.755131 15640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/wal
uuid: "9b87c2b5a3a341998b50fb0c72268505"
format_stamp: "Formatted at 2026-09-26 06:27:31 on dist-test-slave-fkbn"
server_key: "de00aebf9f10e28c919b7cbf00348521"
server_key_iv: "f4603ab8fb374c87285993a6e922d3ba"
server_key_version: "encryptionkey@0"
I20260926 06:27:31.755218 15640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:31.759804 15633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } attempt: 2
I20260926 06:27:31.760159 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.760040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50204)
0926 06:27:31.760095 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.760132 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 06:27:31.760480 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.760396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50214)
0926 06:27:31.760437 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.760459 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 06:27:31.760567 15633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:36971: Network error: Client connection negotiation failed: client connection to 127.2.39.252:36971: connect: Connection refused (error 111)
I20260926 06:27:31.771495 15640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:31.773859 15640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:31.774013 15640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:31.777868 15640 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.252:36971
I20260926 06:27:31.777901 15707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.252:36971 every 8 connection(s)
I20260926 06:27:31.778283 15640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
I20260926 06:27:31.779109 15708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:31.780614 15708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:31.786443 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 15640
I20260926 06:27:31.786558 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790403899360596-2207-0/minicluster-data/master-2/wal/instance
I20260926 06:27:31.786751 2207 external_mini_cluster.cc:1428] Setting key f42a8495b53ac8a6bbb156952a1eaf0b
I20260926 06:27:31.789131 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.781315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.2.39.254:39749)
0926 06:27:31.781376 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.781378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.781530 (+ 152us) server_negotiation.cc:407] Connection header received
0926 06:27:31.782004 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.782005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.782013 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.782067 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.782599 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.783134 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.784042 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.784225 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.784279 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.786221 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.786222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.786222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.786228 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.787714 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.788187 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.788188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.788189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.788252 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.788480 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.788481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.788482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.788625 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.788700 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.789007 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.789070 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 06:27:31.789317 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.781252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:39749 (local address 127.0.0.1:60310)
0926 06:27:31.781405 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.781446 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:31.781916 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:31.782117 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:31.782120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:31.782345 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:31.782527 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.782531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.783219 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.783220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.783963 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.783965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.784297 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.784298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.784397 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.784786 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:31.784795 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:31.786111 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:31.787784 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.787787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.787789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.788098 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.788323 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.788323 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.788323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.788415 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.788694 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:31.788698 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:31.788858 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 06:27:31.788969 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 06:27:31.789112 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260926 06:27:31.789870 15708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
I20260926 06:27:31.796370 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.787157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.2.39.252:36971)
0926 06:27:31.787295 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.787299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.787889 (+ 590us) server_negotiation.cc:407] Connection header received
0926 06:27:31.787911 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.787913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.787926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.788044 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.788746 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.789817 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.790634 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.790937 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.791030 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.793313 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.793321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.793321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.793327 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.795187 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.795555 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.795557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.795558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.795618 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.795816 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.795817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.795817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.795954 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.796091 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.796181 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.796255 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260926 06:27:31.796586 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.790178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.2.39.253:43279)
0926 06:27:31.790245 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.790247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.790256 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:27:31.790280 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.790281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.790290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.790345 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.790600 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.791047 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.791717 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.791935 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.792030 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.794207 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.794208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.794208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.794216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.795494 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.795828 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.795829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.795830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.795884 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.796085 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.796085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.796086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.796197 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.796271 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.796420 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.796501 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:27:31.796584 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.790104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:43279 (local address 127.0.0.1:51776)
0926 06:27:31.790150 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.790159 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:31.790217 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:31.790349 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:31.790350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:31.790476 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:31.790519 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.790521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.791123 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.791124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.791606 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.791607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.792023 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.792024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.792089 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.793006 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:31.793011 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:31.794122 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:31.795593 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.795593 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.795594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.795756 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.795956 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.795957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.795958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.796012 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.796265 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:31.796266 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:31.796307 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 06:27:31.796394 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 06:27:31.796491 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 06:27:31.796878 15708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:31.804088 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.797165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50232 (local address 127.2.39.252:36971)
0926 06:27:31.797197 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.797198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.797209 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:27:31.797276 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.797276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.797285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.797336 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.797605 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.798327 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.799122 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.799418 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.799501 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.801510 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.801511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.801511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.801518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.802897 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.803303 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.803304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.803305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.803362 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.803577 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.803577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.803578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.803670 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.803721 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.803874 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.803982 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":19}
I20260926 06:27:31.804095 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.797121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50232)
0926 06:27:31.797170 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.797180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:31.797223 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:31.797339 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:31.797339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:31.797475 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:31.797506 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.797508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.798392 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.798393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.799051 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.799053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.799513 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.799514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.799561 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.799941 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:31.799945 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:31.801405 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:31.802986 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.802987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.802988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.803204 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.803432 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.803432 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.803433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.803511 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.803735 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:31.803736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:31.803778 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 06:27:31.803849 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 06:27:31.803940 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 06:27:31.805243 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.797434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.2.39.254:39749)
0926 06:27:31.797471 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.797473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.797506 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:27:31.797524 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.797525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.797534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.797603 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.798133 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.798743 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.799729 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.800019 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.800071 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.802558 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.802560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.802560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.802566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.804203 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.804517 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.804518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.804518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.804568 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.804773 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.804773 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.804774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.804890 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.805059 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.805101 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.805167 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:27:31.805987 15708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505: Bootstrap starting.
I20260926 06:27:31.806813 15708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:31.807094 15708 log.cc:824] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:31.807818 15708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505: No bootstrap required, opened a new log
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
I20260926 06:27:31.809620 15708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } }
I20260926 06:27:31.809772 15708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:31.809813 15708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b87c2b5a3a341998b50fb0c72268505, State: Initialized, Role: FOLLOWER
I20260926 06:27:31.809942 15708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } }
I20260926 06:27:31.810245 15708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:27:31.810212 15714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } } }
I20260926 06:27:31.810299 15714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:27:31.810384 15558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } attempt: 2
W20260926 06:27:31.812108 15725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:27:31.812170 15725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:27:31.813337 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.805880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.2.39.253:43279)
0926 06:27:31.805946 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.805948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.805955 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:27:31.805973 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.805974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.805981 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.806049 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.806487 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.806963 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.807648 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.807843 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.807909 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.810134 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.810136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.810136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.810143 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.811873 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.812226 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.812227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.812228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.812284 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.812539 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.812540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.812541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.812665 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.812840 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.813180 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.813248 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:27:31.814924 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
I20260926 06:27:31.812767 15640 master_runner.cc:186] Error getting master registration for 127.2.39.254:39749: OK, instance_id { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" instance_seqno: 1790404051463084 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:27:31.816253 15640 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:27:31.818527 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.810645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50242)
0926 06:27:31.810732 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.810742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:31.810783 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:31.811124 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:31.811125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:31.811282 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:31.811327 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.811329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.812298 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.812299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.813066 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.813069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.813675 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.813676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.813744 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.814138 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:31.814143 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:31.815596 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:31.817310 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.817311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.817311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.817559 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.817838 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.817839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.817839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.817906 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.818199 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:31.818200 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:31.818245 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 06:27:31.818312 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 06:27:31.818405 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 06:27:31.818861 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.810698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.2.39.252:36971)
0926 06:27:31.810979 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.810981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.810988 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:27:31.811006 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.811007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.811024 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.811115 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.811426 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.812209 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.813157 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.813538 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.813898 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.815694 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.815695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.815696 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.815704 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.817233 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.817705 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.817706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.817706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.817766 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.817975 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.817976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.817977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.818106 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.818170 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.818714 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.818782 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260926 06:27:31.820188 15558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc: Bootstrap starting.
I20260926 06:27:31.820930 15558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:31.821256 15558 log.cc:824] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:31.822476 15558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc: No bootstrap required, opened a new log
I20260926 06:27:31.823917 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.816601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.2.39.254:39749)
0926 06:27:31.816646 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.816648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.816807 (+ 159us) server_negotiation.cc:407] Connection header received
0926 06:27:31.816853 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.816854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.816865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.816934 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.817674 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.818365 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.819370 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.819645 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.819710 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.821144 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.821145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.821146 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.821152 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.822750 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.823115 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.823116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.823117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.823167 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.823365 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.823366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.823366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.823477 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.823566 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.823768 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.823841 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:27:31.824940 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.817450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.2.39.253:43279)
0926 06:27:31.817686 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.817688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.817799 (+ 111us) server_negotiation.cc:407] Connection header received
0926 06:27:31.817903 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.817904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.817914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.817943 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.818678 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.819308 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.820112 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.820394 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.820453 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.821865 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.821866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.821866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.821873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.823701 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.824101 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.824102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.824103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.824156 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.824357 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.824357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.824358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.824474 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.824566 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.824770 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.824843 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260926 06:27:31.825682 15558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } }
I20260926 06:27:31.825816 15558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:31.825852 15558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99cde807dceb48aa8b603c977efce5fc, State: Initialized, Role: FOLLOWER
I20260926 06:27:31.825990 15558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } }
I20260926 06:27:31.826284 15735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } } }
I20260926 06:27:31.826473 15735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:27:31.826800 15558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [sys.catalog]: configured and running, proceeding with master startup.
W20260926 06:27:31.828684 15746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:27:31.828734 15746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:27:31.831111 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.820271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.2.39.252:36971)
0926 06:27:31.820302 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.820303 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.824174 (+ 3871us) server_negotiation.cc:407] Connection header received
0926 06:27:31.824260 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.824262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.824271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.824299 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.824841 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.825502 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.826401 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.826690 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.826731 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.828132 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.828133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.828134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.828140 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.829858 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.830309 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.830310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.830311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.830370 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.830596 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.830597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.830597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.830719 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.830813 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.830958 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.831028 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:27:31.829380 15490 master_runner.cc:186] Error getting master registration for 127.2.39.253:43279: OK, instance_id { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" instance_seqno: 1790404051611438 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:27:31.831811 15490 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:27:31.851708 15633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } attempt: 3
Sep 26 06:27:31 dist-test-slave-fkbn krb5kdc[15477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404051, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
I20260926 06:27:31.860416 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.852005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:36971 (local address 127.0.0.1:50254)
0926 06:27:31.852053 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:31.852064 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:31.852100 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:31.852250 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:31.852251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:31.852447 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:31.852501 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.852503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.853445 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.853446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.854327 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:31.854330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.854823 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.854824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:31.854916 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.855423 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:31.855428 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:31.856962 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:31.859015 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.859016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.859016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.859237 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.859726 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:31.859727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:31.859727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:31.859795 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:31.860087 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:31.860088 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:31.860139 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 06:27:31.860215 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 06:27:31.860311 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 06:27:31.860435 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:31.852061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.2.39.252:36971)
0926 06:27:31.852101 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:31.852103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:31.852111 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:27:31.852158 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:31.852159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:31.852169 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:31.852233 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:31.852585 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.853365 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.854422 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:31.854737 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:31.854805 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:31.857050 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:31.857052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:31.857052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:31.857059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:31.858922 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.859311 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.859312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.859313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.859393 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:31.859872 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:31.859873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:31.859874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:31.860011 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:31.860078 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:31.860258 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 06:27:31.860337 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:27:31.862350 15633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2: Bootstrap starting.
I20260926 06:27:31.863095 15633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:31.863405 15633 log.cc:824] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:31.864300 15633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2: No bootstrap required, opened a new log
I20260926 06:27:31.866307 15633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } }
I20260926 06:27:31.866447 15633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:31.866483 15633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75fd290aed2e4c6eb4ba07d47c7d28e2, State: Initialized, Role: FOLLOWER
I20260926 06:27:31.866607 15633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } }
I20260926 06:27:31.866911 15633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:27:31.866874 15749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } } }
I20260926 06:27:31.867087 15749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:27:31.869855 15760 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:27:31.869907 15760 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:27:31.870844 15565 master_runner.cc:263] No leader master found from master 75fd290aed2e4c6eb4ba07d47c7d28e2
I20260926 06:27:31.870894 15565 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 06:27:31.961087 15511 tablet.cc:2369] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:27:32.024591 15661 tablet.cc:2369] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:27:32.061973 15714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:27:32.062086 15714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } }
I20260926 06:27:32.062346 15714 leader_election.cc:288] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99cde807dceb48aa8b603c977efce5fc (127.2.39.254:39749), 75fd290aed2e4c6eb4ba07d47c7d28e2 (127.2.39.253:43279)
I20260926 06:27:32.062592 15607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b87c2b5a3a341998b50fb0c72268505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" is_pre_election: true
I20260926 06:27:32.062600 15532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b87c2b5a3a341998b50fb0c72268505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99cde807dceb48aa8b603c977efce5fc" is_pre_election: true
I20260926 06:27:32.062708 15607 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b87c2b5a3a341998b50fb0c72268505 in term 0.
I20260926 06:27:32.062768 15532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b87c2b5a3a341998b50fb0c72268505 in term 0.
I20260926 06:27:32.062922 15657 leader_election.cc:302] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75fd290aed2e4c6eb4ba07d47c7d28e2, 9b87c2b5a3a341998b50fb0c72268505; no voters:
I20260926 06:27:32.063120 15714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:27:32.063184 15714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:27:32.063200 15714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:32.063915 15714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } }
I20260926 06:27:32.064039 15714 leader_election.cc:288] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [CANDIDATE]: Term 1 election: Requested vote from peers 99cde807dceb48aa8b603c977efce5fc (127.2.39.254:39749), 75fd290aed2e4c6eb4ba07d47c7d28e2 (127.2.39.253:43279)
I20260926 06:27:32.064168 15532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b87c2b5a3a341998b50fb0c72268505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99cde807dceb48aa8b603c977efce5fc"
I20260926 06:27:32.064200 15607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9b87c2b5a3a341998b50fb0c72268505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2"
I20260926 06:27:32.064272 15607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:32.064272 15532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:32.064929 15607 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b87c2b5a3a341998b50fb0c72268505 in term 1.
I20260926 06:27:32.065052 15532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b87c2b5a3a341998b50fb0c72268505 in term 1.
I20260926 06:27:32.065121 15657 leader_election.cc:302] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75fd290aed2e4c6eb4ba07d47c7d28e2, 9b87c2b5a3a341998b50fb0c72268505; no voters:
I20260926 06:27:32.065232 15714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:27:32.065368 15714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [term 1 LEADER]: Becoming Leader. State: Replica: 9b87c2b5a3a341998b50fb0c72268505, State: Running, Role: LEADER
I20260926 06:27:32.065505 15714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } }
I20260926 06:27:32.066012 15762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b87c2b5a3a341998b50fb0c72268505. Latest consensus state: current_term: 1 leader_uuid: "9b87c2b5a3a341998b50fb0c72268505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } } }
I20260926 06:27:32.066114 15762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:32.066301 15764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:27:32.066504 15764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:27:32.067983 15607 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [term 1 FOLLOWER]: Refusing update from remote peer 9b87c2b5a3a341998b50fb0c72268505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:27:32.068001 15532 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [term 1 FOLLOWER]: Refusing update from remote peer 9b87c2b5a3a341998b50fb0c72268505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:27:32.068300 15714 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:27:32.068457 15762 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:27:32.069628 15735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b87c2b5a3a341998b50fb0c72268505. Latest consensus state: current_term: 1 leader_uuid: "9b87c2b5a3a341998b50fb0c72268505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } } }
I20260926 06:27:32.069636 15749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b87c2b5a3a341998b50fb0c72268505. Latest consensus state: current_term: 1 leader_uuid: "9b87c2b5a3a341998b50fb0c72268505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } } }
I20260926 06:27:32.069736 15749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:27:32.069738 15735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:27:32.070677 15714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b87c2b5a3a341998b50fb0c72268505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } } }
I20260926 06:27:32.070752 15714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:32.070848 15735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b87c2b5a3a341998b50fb0c72268505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } } }
I20260926 06:27:32.070930 15735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:27:32.070909 15762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9b87c2b5a3a341998b50fb0c72268505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } } }
I20260926 06:27:32.070976 15762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:32.071071 15764 catalog_manager.cc:1347] Generated new cluster ID: cfa490d6274d4cb9bf37704daa02c62b
I20260926 06:27:32.071113 15764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:27:32.071192 15766 mvcc.cc:205] Tried to move back new op lower bound from 7333494997267935232 to 7333494997261049856. Current Snapshot: MvccSnapshot[applied={T|T < 7333494997267935232}]
I20260926 06:27:32.071305 15749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9b87c2b5a3a341998b50fb0c72268505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99cde807dceb48aa8b603c977efce5fc" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 39749 } } peers { permanent_uuid: "75fd290aed2e4c6eb4ba07d47c7d28e2" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 43279 } } peers { permanent_uuid: "9b87c2b5a3a341998b50fb0c72268505" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 36971 } } }
I20260926 06:27:32.071405 15749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:27:32.081635 15764 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:27:32.082240 15764 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:27:32.087343 15764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b87c2b5a3a341998b50fb0c72268505: Generated new TSK 0
I20260926 06:27:32.087519 15764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:27:32.829154 15746 catalog_manager.cc:1259] Loaded cluster ID: cfa490d6274d4cb9bf37704daa02c62b
I20260926 06:27:32.829232 15746 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc: loading cluster ID for follower catalog manager: success
I20260926 06:27:32.830025 15746 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc: acquiring CA information for follower catalog manager: success
I20260926 06:27:32.830277 15746 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99cde807dceb48aa8b603c977efce5fc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:27:32.870457 15760 catalog_manager.cc:1259] Loaded cluster ID: cfa490d6274d4cb9bf37704daa02c62b
I20260926 06:27:32.870535 15760 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2: loading cluster ID for follower catalog manager: success
I20260926 06:27:32.871328 15760 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2: acquiring CA information for follower catalog manager: success
I20260926 06:27:32.871584 15760 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 75fd290aed2e4c6eb4ba07d47c7d28e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:27:33.145561 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 15490
I20260926 06:27:33.150960 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 15565
I20260926 06:27:33.155583 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 15640
2026-09-26T06:27:33Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2958 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:27:33 dist-test-slave-fkbn krb5kdc[15779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:27:33 dist-test-slave-fkbn krb5kdc[15779](info): set up 2 sockets
Sep 26 06:27:33 dist-test-slave-fkbn krb5kdc[15779](info): commencing operation
krb5kdc: starting...
W20260926 06:27:34.189656 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:27:34 dist-test-slave-fkbn krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:27:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:27:34.306134 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36479
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:36511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:36479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:34.385308 15792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:34.385540 15792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:34.385577 15792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:34.386736 15792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:27:34.386802 15792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:34.386829 15792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:34.386858 15792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:27:34.386890 15792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:27:34.388300 15792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:36511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:36479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:34.388573 15792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:34.388787 15792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:34.393594 15800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:34.393647 15798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:34.393704 15792 server_base.cc:1034] running on GCE node
W20260926 06:27:34.393536 15797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:34.394022 15792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:34.394291 15792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:34.395459 15792 hybrid_clock.cc:648] HybridClock initialized: now 1790404054395444 us; error 45 us; skew 500 ppm
Sep 26 06:27:34 dist-test-slave-fkbn krb5kdc[15779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404054, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:34.397845 15792 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:27:34.398326 15792 webserver.cc:460] Webserver started at http://127.2.39.254:38731/ using document root <none> and password file <none>
I20260926 06:27:34.398667 15792 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:34.398746 15792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:34.398912 15792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:34.399958 15792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "22829de6b1b946e3a1e1b1fd9b613c37"
format_stamp: "Formatted at 2026-09-26 06:27:34 on dist-test-slave-fkbn"
server_key: "f0fca1971e7414c81d98a577fa3585e5"
server_key_iv: "9ac4799522c584c20b6df4ad3a9a541d"
server_key_version: "encryptionkey@0"
I20260926 06:27:34.400334 15792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "22829de6b1b946e3a1e1b1fd9b613c37"
format_stamp: "Formatted at 2026-09-26 06:27:34 on dist-test-slave-fkbn"
server_key: "f0fca1971e7414c81d98a577fa3585e5"
server_key_iv: "9ac4799522c584c20b6df4ad3a9a541d"
server_key_version: "encryptionkey@0"
I20260926 06:27:34.401667 15792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:27:34.402387 15807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:34.402549 15792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:34.402638 15792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "22829de6b1b946e3a1e1b1fd9b613c37"
format_stamp: "Formatted at 2026-09-26 06:27:34 on dist-test-slave-fkbn"
server_key: "f0fca1971e7414c81d98a577fa3585e5"
server_key_iv: "9ac4799522c584c20b6df4ad3a9a541d"
server_key_version: "encryptionkey@0"
I20260926 06:27:34.402719 15792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:34.420909 15792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:34.422732 15792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:34.422899 15792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:34.426949 15792 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:36479
I20260926 06:27:34.426982 15859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:36479 every 8 connection(s)
I20260926 06:27:34.427385 15792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:27:34.428217 15860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:34.430204 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 15792
I20260926 06:27:34.430310 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:27:34.430495 2207 external_mini_cluster.cc:1428] Setting key dad68bbd345e3ee237b28f5dd01fafcf
I20260926 06:27:34.430670 15860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37: Bootstrap starting.
I20260926 06:27:34.431345 15860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:34.431659 15860 log.cc:824] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:34.432320 15860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37: No bootstrap required, opened a new log
I20260926 06:27:34.434141 15860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22829de6b1b946e3a1e1b1fd9b613c37" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36479 } }
I20260926 06:27:34.434289 15860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:34.434333 15860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22829de6b1b946e3a1e1b1fd9b613c37, State: Initialized, Role: FOLLOWER
I20260926 06:27:34.434465 15860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22829de6b1b946e3a1e1b1fd9b613c37" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36479 } }
I20260926 06:27:34.434542 15860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:27:34.434583 15860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:27:34.434629 15860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:34.435143 15860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22829de6b1b946e3a1e1b1fd9b613c37" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36479 } }
I20260926 06:27:34.435267 15860 leader_election.cc:302] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22829de6b1b946e3a1e1b1fd9b613c37; no voters:
I20260926 06:27:34.435420 15860 leader_election.cc:288] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:27:34.435559 15865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:27:34.435671 15865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [term 1 LEADER]: Becoming Leader. State: Replica: 22829de6b1b946e3a1e1b1fd9b613c37, State: Running, Role: LEADER
I20260926 06:27:34.435690 15860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:27:34.435751 15865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22829de6b1b946e3a1e1b1fd9b613c37" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36479 } }
I20260926 06:27:34.436088 15865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22829de6b1b946e3a1e1b1fd9b613c37. Latest consensus state: current_term: 1 leader_uuid: "22829de6b1b946e3a1e1b1fd9b613c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22829de6b1b946e3a1e1b1fd9b613c37" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36479 } } }
I20260926 06:27:34.436100 15866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22829de6b1b946e3a1e1b1fd9b613c37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22829de6b1b946e3a1e1b1fd9b613c37" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36479 } } }
I20260926 06:27:34.436184 15865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:34.436216 15866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37 [sys.catalog]: This master's current role is: LEADER
Sep 26 06:27:34 dist-test-slave-fkbn krb5kdc[15779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
W20260926 06:27:34.437472 15881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:27:34.437521 15881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:27:34.437590 15873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:27:34.437769 15873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:27:34.439349 15873 catalog_manager.cc:1347] Generated new cluster ID: fc8722e75d6c45ef9e0ef9a6c33dc0d8
I20260926 06:27:34.439404 15873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:27:34.441540 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:34.430967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.2.39.254:36479)
0926 06:27:34.431173 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:34.431184 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:34.431214 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:27:34.431685 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:34.431692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:34.431772 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:34.431939 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:34.432615 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:34.433477 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:34.434259 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:34.434841 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:34.434962 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:34.437935 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:34.437939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:34.437939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:34.437950 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:34.439945 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:34.440318 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:34.440321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:34.440322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:34.440428 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:34.440632 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:34.440633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:34.440634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:34.440939 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:34.441119 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:34.441194 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 06:27:34.441335 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
I20260926 06:27:34.442405 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:27:34.461711 15873 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:27:34.462451 15873 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:27:34.471338 15873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22829de6b1b946e3a1e1b1fd9b613c37: Generated new TSK 0
I20260926 06:27:34.471529 15873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:27:34.496353 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 15792
2026-09-26T06:27:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1339 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:27:34 dist-test-slave-fkbn krb5kdc[15887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:27:34 dist-test-slave-fkbn krb5kdc[15887](info): set up 2 sockets
Sep 26 06:27:34 dist-test-slave-fkbn krb5kdc[15887](info): commencing operation
krb5kdc: starting...
W20260926 06:27:35.541553 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:27:35 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:27:35.664808 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:33781
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:33781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:35.748608 15900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:35.748845 15900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:35.748883 15900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:35.750036 15900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:27:35.750113 15900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:35.750144 15900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:35.750165 15900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:27:35.750193 15900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:27:35.751626 15900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:33781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:33781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:35.751901 15900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:35.752115 15900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:35.756850 15905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:35.756896 15908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:35.756893 15906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:35.756918 15900 server_base.cc:1034] running on GCE node
I20260926 06:27:35.757257 15900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:35.757490 15900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:35.758630 15900 hybrid_clock.cc:648] HybridClock initialized: now 1790404055758609 us; error 37 us; skew 500 ppm
Sep 26 06:27:35 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:35.760989 15900 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:27:35.761418 15900 webserver.cc:460] Webserver started at http://127.2.39.254:43901/ using document root <none> and password file <none>
I20260926 06:27:35.761642 15900 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:35.761687 15900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:35.761773 15900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:35.762727 15900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "98ad1b2458984a8bb885c7faa50a7bd6"
format_stamp: "Formatted at 2026-09-26 06:27:35 on dist-test-slave-fkbn"
server_key: "19c3b7d45954d53a446c880c4855c190"
server_key_iv: "eeec277243972390afc8867dbbf41d08"
server_key_version: "encryptionkey@0"
I20260926 06:27:35.763046 15900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "98ad1b2458984a8bb885c7faa50a7bd6"
format_stamp: "Formatted at 2026-09-26 06:27:35 on dist-test-slave-fkbn"
server_key: "19c3b7d45954d53a446c880c4855c190"
server_key_iv: "eeec277243972390afc8867dbbf41d08"
server_key_version: "encryptionkey@0"
I20260926 06:27:35.764298 15900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:27:35.765118 15915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:35.765290 15900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:27:35.765388 15900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "98ad1b2458984a8bb885c7faa50a7bd6"
format_stamp: "Formatted at 2026-09-26 06:27:35 on dist-test-slave-fkbn"
server_key: "19c3b7d45954d53a446c880c4855c190"
server_key_iv: "eeec277243972390afc8867dbbf41d08"
server_key_version: "encryptionkey@0"
I20260926 06:27:35.765461 15900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:35.790287 15900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:35.792065 15900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:35.792219 15900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:35.796109 15900 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:33781
I20260926 06:27:35.796144 15967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:33781 every 8 connection(s)
I20260926 06:27:35.796490 15900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:27:35.797423 15968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:35.799145 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 15900
I20260926 06:27:35.799258 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:27:35.799440 2207 external_mini_cluster.cc:1428] Setting key 33e99dfe737eff106e46a226627febba
I20260926 06:27:35.800154 15968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Bootstrap starting.
I20260926 06:27:35.800719 15968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:35.801079 15968 log.cc:824] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:35.801836 15968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: No bootstrap required, opened a new log
I20260926 06:27:35.804200 15968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:35.804404 15968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:35.804454 15968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ad1b2458984a8bb885c7faa50a7bd6, State: Initialized, Role: FOLLOWER
I20260926 06:27:35.804591 15968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:35.804661 15968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:27:35.804700 15968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:27:35.804745 15968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:35.805500 15968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:35.805630 15968 leader_election.cc:302] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ad1b2458984a8bb885c7faa50a7bd6; no voters:
I20260926 06:27:35.805797 15968 leader_election.cc:288] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:27:35.805833 15973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:27:35.805994 15973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 1 LEADER]: Becoming Leader. State: Replica: 98ad1b2458984a8bb885c7faa50a7bd6, State: Running, Role: LEADER
I20260926 06:27:35.806061 15968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 06:27:35 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:35.806555 15973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:35.806906 15975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } } }
I20260926 06:27:35.806914 15974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ad1b2458984a8bb885c7faa50a7bd6. Latest consensus state: current_term: 1 leader_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } } }
I20260926 06:27:35.806993 15975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:35.807001 15974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:35.807205 15980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:27:35.808310 15980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:27:35.809806 15980 catalog_manager.cc:1347] Generated new cluster ID: 0962f51df9d646a78e36364678855d8d
I20260926 06:27:35.809859 15980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:27:35.810766 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:35.799945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58048 (local address 127.2.39.254:33781)
0926 06:27:35.800123 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:35.800131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:35.800155 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:27:35.800607 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:35.800613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:35.800658 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:35.800769 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:35.801553 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:35.802713 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:35.803798 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:35.804193 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:35.804272 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:35.807121 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:35.807125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:35.807125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:35.807138 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:35.809204 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:35.809616 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:35.809619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:35.809620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:35.809731 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:35.809941 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:35.809942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:35.809942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:35.810167 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:35.810310 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:35.810425 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 06:27:35.810573 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260926 06:27:35.819335 15980 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:27:35.820044 15980 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:27:35.824816 15980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Generated new TSK 0
I20260926 06:27:35.824993 15980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:27:35.858004 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33781
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:35.939461 15994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:35.939663 15994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:35.939688 15994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:35.940841 15994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:35.940883 15994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:35.940909 15994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:27:35.942487 15994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:35.942734 15994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:35.942968 15994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:35.947999 16000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:35.948009 15999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:35.948009 16002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:35.948076 15994 server_base.cc:1034] running on GCE node
I20260926 06:27:35.948410 15994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:35.948691 15994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:35.949848 15994 hybrid_clock.cc:648] HybridClock initialized: now 1790404055949841 us; error 34 us; skew 500 ppm
Sep 26 06:27:35 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:35.952275 15994 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:27:35.952701 15994 webserver.cc:460] Webserver started at http://127.2.39.193:42053/ using document root <none> and password file <none>
I20260926 06:27:35.952891 15994 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:35.952931 15994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:35.953017 15994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:35.954113 15994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "3423711e5d5c456abfe9e34a5713238d"
format_stamp: "Formatted at 2026-09-26 06:27:35 on dist-test-slave-fkbn"
server_key: "ad09938c5880e0828629241f09f6a5b7"
server_key_iv: "45b833fa231f5409fd1e1bec93366d50"
server_key_version: "encryptionkey@0"
I20260926 06:27:35.954455 15994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "3423711e5d5c456abfe9e34a5713238d"
format_stamp: "Formatted at 2026-09-26 06:27:35 on dist-test-slave-fkbn"
server_key: "ad09938c5880e0828629241f09f6a5b7"
server_key_iv: "45b833fa231f5409fd1e1bec93366d50"
server_key_version: "encryptionkey@0"
I20260926 06:27:35.955715 15994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:27:35.956365 16009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:35.956540 15994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:35.956619 15994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "3423711e5d5c456abfe9e34a5713238d"
format_stamp: "Formatted at 2026-09-26 06:27:35 on dist-test-slave-fkbn"
server_key: "ad09938c5880e0828629241f09f6a5b7"
server_key_iv: "45b833fa231f5409fd1e1bec93366d50"
server_key_version: "encryptionkey@0"
I20260926 06:27:35.956672 15994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:35.988546 15994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:35.990891 15994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:35.991017 15994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:35.991199 15994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:35.991617 15994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:35.991660 15994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:35.991683 15994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:35.991695 15994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:35.997529 15994 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:45395
I20260926 06:27:35.997643 16122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:45395 every 8 connection(s)
I20260926 06:27:35.997959 15994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:27:36.002894 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 15994
I20260926 06:27:36.003017 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:27:36.003198 2207 external_mini_cluster.cc:1428] Setting key 8723b9a672aacaa8ac030e3523dc8f9d
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404055, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:36.007748 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:35.998661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:38805 (local address 127.2.39.254:33781)
0926 06:27:35.998742 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:35.998744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:35.998919 (+ 175us) server_negotiation.cc:407] Connection header received
0926 06:27:35.999429 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:35.999430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:35.999440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:35.999495 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:36.000405 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.000891 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.001981 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.002183 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.002239 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.004502 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:36.004503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:36.004504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:36.004510 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:36.006189 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.006673 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.006674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.006675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.006733 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.007035 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.007037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.007037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.007152 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:36.007232 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.007551 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.007652 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:27:36.007931 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:35.998627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:38805)
0926 06:27:35.998786 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:35.998821 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:35.999311 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:35.999551 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:35.999555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.000129 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:36.000325 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.000329 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.000985 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.000986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.001889 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.001892 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.002265 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.002266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.002373 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.002869 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:36.002881 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:36.004399 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:36.006295 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.006302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.006305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.006591 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.006797 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.006798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.006798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.006962 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.007257 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:36.007263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:36.007410 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.007528 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.007690 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260926 06:27:36.008466 16123 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33781
I20260926 06:27:36.008565 16123 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:36.008771 16123 heartbeater.cc:511] Master 127.2.39.254:33781 requested a full tablet report, sending...
I20260926 06:27:36.009366 15932 ts_manager.cc:194] Registered new tserver with Master: 3423711e5d5c456abfe9e34a5713238d (127.2.39.193:45395)
I20260926 06:27:36.010344 15932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:38805
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:27:36.029019 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33781
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:36.111877 16128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:36.112085 16128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:36.112109 16128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:36.113237 16128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:36.113291 16128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:36.113318 16128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:27:36.114722 16128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:36.114918 16128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:36.115109 16128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:36.120160 16133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:36.120160 16134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:36.120174 16136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:36.120250 16128 server_base.cc:1034] running on GCE node
I20260926 06:27:36.120627 16128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:36.120860 16128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:36.122012 16128 hybrid_clock.cc:648] HybridClock initialized: now 1790404056122005 us; error 34 us; skew 500 ppm
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:36.124362 16128 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:27:36.124745 16128 webserver.cc:460] Webserver started at http://127.2.39.194:43465/ using document root <none> and password file <none>
I20260926 06:27:36.124919 16128 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:36.124957 16128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:36.125048 16128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:36.126108 16128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac"
format_stamp: "Formatted at 2026-09-26 06:27:36 on dist-test-slave-fkbn"
server_key: "1000294db1a88ba7b89448a73db137fc"
server_key_iv: "c13fb4c662f902b0748eb9abab89bbcc"
server_key_version: "encryptionkey@0"
I20260926 06:27:36.126401 16128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac"
format_stamp: "Formatted at 2026-09-26 06:27:36 on dist-test-slave-fkbn"
server_key: "1000294db1a88ba7b89448a73db137fc"
server_key_iv: "c13fb4c662f902b0748eb9abab89bbcc"
server_key_version: "encryptionkey@0"
I20260926 06:27:36.127636 16128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:27:36.128254 16143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.128401 16128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:36.128490 16128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac"
format_stamp: "Formatted at 2026-09-26 06:27:36 on dist-test-slave-fkbn"
server_key: "1000294db1a88ba7b89448a73db137fc"
server_key_iv: "c13fb4c662f902b0748eb9abab89bbcc"
server_key_version: "encryptionkey@0"
I20260926 06:27:36.128544 16128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:36.153209 16128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:36.155524 16128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:36.155632 16128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:36.155821 16128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:36.156082 16128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:36.156114 16128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.156136 16128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:36.156149 16128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.161876 16128 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:34313
I20260926 06:27:36.161916 16256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:34313 every 8 connection(s)
I20260926 06:27:36.162237 16128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:27:36.163938 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 16128
I20260926 06:27:36.164032 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:27:36.164211 2207 external_mini_cluster.cc:1428] Setting key 3a2a03679b82a18d92be628d179b1dd6
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
I20260926 06:27:36.173127 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.162914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:37585 (local address 127.2.39.254:33781)
0926 06:27:36.162992 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.162995 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.163254 (+ 259us) server_negotiation.cc:407] Connection header received
0926 06:27:36.163757 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.163758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.163783 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.163848 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:36.164986 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.165797 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.167059 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.167396 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.167457 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.170282 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:36.170283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:36.170283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:36.170289 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:36.171619 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.172076 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.172077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.172078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.172128 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.172472 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.172472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.172473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.172581 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:36.172633 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.172948 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.173038 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 06:27:36.173384 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.162879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:37585)
0926 06:27:36.163080 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.163148 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.163661 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.163898 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.163902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.164643 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:36.164904 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.164910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.165879 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.165880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.166962 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.166964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.167483 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.167485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.167648 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.168183 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:36.168193 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:36.170190 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:36.171699 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.171706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.171708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.172002 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.172223 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.172224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.172224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.172398 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.172644 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:36.172650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:36.172815 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.172931 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.173114 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260926 06:27:36.173888 16257 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33781
I20260926 06:27:36.173987 16257 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:36.174191 16257 heartbeater.cc:511] Master 127.2.39.254:33781 requested a full tablet report, sending...
I20260926 06:27:36.174556 15932 ts_manager.cc:194] Registered new tserver with Master: abe7fcf398a9465c8bbfbe9e9cc323ac (127.2.39.194:34313)
I20260926 06:27:36.174924 15932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:37585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:27:36.190742 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33781
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:36.271617 16262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:36.271813 16262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:36.271836 16262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:36.272962 16262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:36.273007 16262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:36.273041 16262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:27:36.274518 16262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:36.274796 16262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:36.275022 16262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:36.279841 16268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:36.279841 16267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:36.279865 16270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:36.280241 16262 server_base.cc:1034] running on GCE node
I20260926 06:27:36.280452 16262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:36.280689 16262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:36.281847 16262 hybrid_clock.cc:648] HybridClock initialized: now 1790404056281841 us; error 36 us; skew 500 ppm
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:36.284363 16262 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:27:36.284853 16262 webserver.cc:460] Webserver started at http://127.2.39.195:37261/ using document root <none> and password file <none>
I20260926 06:27:36.285050 16262 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:36.285090 16262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:36.285180 16262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:36.286295 16262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "128bd3a164cc460bad90618cdd643b70"
format_stamp: "Formatted at 2026-09-26 06:27:36 on dist-test-slave-fkbn"
server_key: "5038c868c075fab04211736d75f44adc"
server_key_iv: "73b5a9522453b26ffd25b26ded4b301c"
server_key_version: "encryptionkey@0"
I20260926 06:27:36.286618 16262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "128bd3a164cc460bad90618cdd643b70"
format_stamp: "Formatted at 2026-09-26 06:27:36 on dist-test-slave-fkbn"
server_key: "5038c868c075fab04211736d75f44adc"
server_key_iv: "73b5a9522453b26ffd25b26ded4b301c"
server_key_version: "encryptionkey@0"
I20260926 06:27:36.287976 16262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:27:36.288653 16277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.288815 16262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:36.288883 16262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "128bd3a164cc460bad90618cdd643b70"
format_stamp: "Formatted at 2026-09-26 06:27:36 on dist-test-slave-fkbn"
server_key: "5038c868c075fab04211736d75f44adc"
server_key_iv: "73b5a9522453b26ffd25b26ded4b301c"
server_key_version: "encryptionkey@0"
I20260926 06:27:36.288935 16262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:36.338893 16262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:36.340684 16262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:36.340795 16262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:36.340979 16262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:36.341246 16262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:36.341277 16262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.341298 16262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:36.341311 16262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.346961 16262 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:44687
I20260926 06:27:36.347009 16390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:44687 every 8 connection(s)
I20260926 06:27:36.347337 16262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:36.355621 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 16262
I20260926 06:27:36.355727 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:27:36.355885 2207 external_mini_cluster.cc:1428] Setting key 7a12e242ea5fd09a683b59475fde60f6
I20260926 06:27:36.356251 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.348051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:60047 (local address 127.2.39.254:33781)
0926 06:27:36.348115 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.348118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.348301 (+ 183us) server_negotiation.cc:407] Connection header received
0926 06:27:36.348833 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.348834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.348847 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.348902 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:36.349896 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.350410 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.351386 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.351588 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.351660 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.353731 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:36.353733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:36.353733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:36.353740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:36.354873 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.355282 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.355284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.355284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.355340 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.355573 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.355574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.355575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.355687 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:36.355752 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.356074 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.356162 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:27:36.356446 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.348035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:60047)
0926 06:27:36.348185 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.348220 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.348730 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.348973 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.348977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.349551 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:36.349803 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.349808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.350505 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.350507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.351297 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.351299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.351678 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.351679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.351798 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.352284 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:36.352292 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:36.353640 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:36.354957 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.354962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.354964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.355197 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.355401 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.355402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.355403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.355510 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.355754 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:36.355761 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:36.355889 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.356023 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.356191 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260926 06:27:36.357028 16391 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33781
I20260926 06:27:36.357123 16391 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:36.357290 16391 heartbeater.cc:511] Master 127.2.39.254:33781 requested a full tablet report, sending...
I20260926 06:27:36.357628 15932 ts_manager.cc:194] Registered new tserver with Master: 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687)
I20260926 06:27:36.358001 15932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:60047
I20260926 06:27:36.367717 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:27:36.374171 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.368954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58056 (local address 127.2.39.254:33781)
0926 06:27:36.368990 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.368992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.369112 (+ 120us) server_negotiation.cc:407] Connection header received
0926 06:27:36.369154 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.369155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.369163 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.369213 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:36.369770 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.370229 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.370886 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.371059 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.371105 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.372097 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:36.372099 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:36.372099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:36.372104 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:36.373235 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.373524 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.373525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.373525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.373624 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.373784 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.373785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.373785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.373908 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:36.373977 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.374055 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.374110 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:27:36.375495 15932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:27:36.376963 15932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:27:36.383338 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.379316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.2.39.194:34313)
0926 06:27:36.379576 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.379581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.379632 (+ 51us) server_negotiation.cc:407] Connection header received
0926 06:27:36.379740 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.379742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.379760 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.379830 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:36.380156 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.380850 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.382166 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.382933 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.383006 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.383063 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.383214 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.383259 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
I20260926 06:27:36.383327 16403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.379142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:34313 (local address 127.0.0.1:51698)
0926 06:27:36.379542 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.379583 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.379652 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.380008 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.380011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.380013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:36.380076 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.380077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.380946 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.380947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.382082 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.382085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.383028 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.383029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.383104 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.383117 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.383190 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.383243 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":58,"threads_started":1}
I20260926 06:27:36.383527 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.379367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.2.39.195:44687)
0926 06:27:36.379544 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.379548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.379993 (+ 445us) server_negotiation.cc:407] Connection header received
0926 06:27:36.380035 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.380037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.380051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.380118 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:36.380216 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.381329 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.382522 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.383010 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.383080 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.383135 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.383367 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.383413 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260926 06:27:36.383813 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.379154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:44687 (local address 127.0.0.1:54732)
0926 06:27:36.379869 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.379876 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.379923 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.380129 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.380130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.380132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:36.380163 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.380164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.381440 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.381441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.382444 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.382445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.383167 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.383168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.383221 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.383224 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.383268 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.383296 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":73,"thread_start_us":73,"threads_started":1}
I20260926 06:27:36.383868 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.379322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.2.39.193:45395)
0926 06:27:36.379491 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.379496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.379876 (+ 380us) server_negotiation.cc:407] Connection header received
0926 06:27:36.379933 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.379935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.379950 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.380041 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:36.380268 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.381340 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.382694 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.383407 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.383475 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.383523 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.383738 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.383784 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260926 06:27:36.383878 16404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.379147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45395 (local address 127.0.0.1:45460)
0926 06:27:36.379750 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.379758 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.379805 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.380128 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.380130 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.380136 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:36.380187 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.380189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.381451 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.381452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.382598 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.382601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.383559 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.383560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.383650 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.383655 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.383707 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.383743 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":7,"thread_start_us":77,"threads_started":1}
I20260926 06:27:36.384436 16325 tablet_service.cc:1467] Processing CreateTablet for tablet 44921cc26dc24fefafd99abe5f3de302 (DEFAULT_TABLE table=test-table [id=b1acf0d71d254623b980c28fb2baea94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:36.384500 16191 tablet_service.cc:1467] Processing CreateTablet for tablet 44921cc26dc24fefafd99abe5f3de302 (DEFAULT_TABLE table=test-table [id=b1acf0d71d254623b980c28fb2baea94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:36.384766 16325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44921cc26dc24fefafd99abe5f3de302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:36.384809 16191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44921cc26dc24fefafd99abe5f3de302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:36.387337 16408 tablet_bootstrap.cc:492] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: Bootstrap starting.
I20260926 06:27:36.387359 16409 tablet_bootstrap.cc:492] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Bootstrap starting.
I20260926 06:27:36.387846 16057 tablet_service.cc:1467] Processing CreateTablet for tablet 44921cc26dc24fefafd99abe5f3de302 (DEFAULT_TABLE table=test-table [id=b1acf0d71d254623b980c28fb2baea94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:36.387984 16409 tablet_bootstrap.cc:654] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:36.388082 16408 tablet_bootstrap.cc:654] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:36.388188 16057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44921cc26dc24fefafd99abe5f3de302. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:36.388497 16409 log.cc:824] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:36.388655 16408 log.cc:824] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:36.389423 16408 tablet_bootstrap.cc:492] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: No bootstrap required, opened a new log
I20260926 06:27:36.389495 16408 ts_tablet_manager.cc:1397] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:27:36.390084 16409 tablet_bootstrap.cc:492] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: No bootstrap required, opened a new log
I20260926 06:27:36.390159 16409 ts_tablet_manager.cc:1397] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 06:27:36.391611 16408 raft_consensus.cc:348] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
I20260926 06:27:36.391775 16408 raft_consensus.cc:374] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:36.391801 16408 raft_consensus.cc:729] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128bd3a164cc460bad90618cdd643b70, State: Initialized, Role: FOLLOWER
I20260926 06:27:36.391758 16409 raft_consensus.cc:348] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
I20260926 06:27:36.391938 16409 raft_consensus.cc:374] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:36.391933 16408 consensus_queue.cc:260] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
I20260926 06:27:36.391989 16409 raft_consensus.cc:729] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe7fcf398a9465c8bbfbe9e9cc323ac, State: Initialized, Role: FOLLOWER
I20260926 06:27:36.392102 16409 consensus_queue.cc:260] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
I20260926 06:27:36.392194 16408 ts_tablet_manager.cc:1428] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:27:36.392213 16391 heartbeater.cc:503] Master 127.2.39.254:33781 was elected leader, sending a full tablet report...
I20260926 06:27:36.392374 16409 ts_tablet_manager.cc:1428] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:27:36.392398 16257 heartbeater.cc:503] Master 127.2.39.254:33781 was elected leader, sending a full tablet report...
I20260926 06:27:36.392390 16412 tablet_bootstrap.cc:492] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Bootstrap starting.
I20260926 06:27:36.393042 16412 tablet_bootstrap.cc:654] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:36.393631 16412 log.cc:824] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:36.394212 16412 tablet_bootstrap.cc:492] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: No bootstrap required, opened a new log
I20260926 06:27:36.394263 16412 ts_tablet_manager.cc:1397] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:27:36.395713 16412 raft_consensus.cc:348] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
I20260926 06:27:36.395845 16412 raft_consensus.cc:374] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:36.395866 16412 raft_consensus.cc:729] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3423711e5d5c456abfe9e34a5713238d, State: Initialized, Role: FOLLOWER
I20260926 06:27:36.395968 16412 consensus_queue.cc:260] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
I20260926 06:27:36.396231 16412 ts_tablet_manager.cc:1428] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:27:36.396245 16123 heartbeater.cc:503] Master 127.2.39.254:33781 was elected leader, sending a full tablet report...
I20260926 06:27:36.397128 16416 raft_consensus.cc:482] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:27:36.397190 16416 raft_consensus.cc:504] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
I20260926 06:27:36.397414 16416 leader_election.cc:288] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abe7fcf398a9465c8bbfbe9e9cc323ac (127.2.39.194:34313), 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687)
I20260926 06:27:36.400087 16406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.397639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:45815 (local address 127.2.39.194:34313)
0926 06:27:36.397682 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.397684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.397726 (+ 42us) server_negotiation.cc:407] Connection header received
0926 06:27:36.397791 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.397792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.397804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.397858 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:36.397994 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.398467 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.399393 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.399791 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.399859 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.399868 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.400007 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.400041 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:27:36.400120 16125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.397562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:34313 (local address 127.2.39.193:45815)
0926 06:27:36.397668 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.397682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.397740 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.397861 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.397862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.397863 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:36.397930 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.397931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.398545 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.398546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.399299 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.399301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.399857 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.399858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.399938 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.399942 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.400003 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.400048 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":21}
I20260926 06:27:36.400384 16407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.397675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:53835 (local address 127.2.39.195:44687)
0926 06:27:36.397726 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.397728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.397845 (+ 117us) server_negotiation.cc:407] Connection header received
0926 06:27:36.397917 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.397918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.397928 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.397998 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:36.398151 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.398590 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.399469 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.400001 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.400060 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.400068 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.400286 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.400323 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:27:36.400390 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.397562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:44687 (local address 127.2.39.193:53835)
0926 06:27:36.397794 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.397803 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.397859 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.398002 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.398002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.398004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:36.398069 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.398070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.398662 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.398663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.399386 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.399388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.400098 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.400099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.400213 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.400217 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.400279 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.400318 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":103,"thread_start_us":85,"threads_started":1}
I20260926 06:27:36.400460 16211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44921cc26dc24fefafd99abe5f3de302" candidate_uuid: "3423711e5d5c456abfe9e34a5713238d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" is_pre_election: true
I20260926 06:27:36.400619 16211 raft_consensus.cc:2454] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3423711e5d5c456abfe9e34a5713238d in term 0.
I20260926 06:27:36.400679 16345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44921cc26dc24fefafd99abe5f3de302" candidate_uuid: "3423711e5d5c456abfe9e34a5713238d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128bd3a164cc460bad90618cdd643b70" is_pre_election: true
I20260926 06:27:36.400833 16345 raft_consensus.cc:2454] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3423711e5d5c456abfe9e34a5713238d in term 0.
I20260926 06:27:36.400866 16010 leader_election.cc:302] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3423711e5d5c456abfe9e34a5713238d, abe7fcf398a9465c8bbfbe9e9cc323ac; no voters:
I20260926 06:27:36.400995 16416 raft_consensus.cc:2781] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:27:36.401050 16416 raft_consensus.cc:482] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:27:36.401081 16416 raft_consensus.cc:3037] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:36.401777 16416 raft_consensus.cc:504] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
I20260926 06:27:36.401901 16416 leader_election.cc:288] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [CANDIDATE]: Term 1 election: Requested vote from peers abe7fcf398a9465c8bbfbe9e9cc323ac (127.2.39.194:34313), 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687)
I20260926 06:27:36.402029 16211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44921cc26dc24fefafd99abe5f3de302" candidate_uuid: "3423711e5d5c456abfe9e34a5713238d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac"
I20260926 06:27:36.402040 16345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44921cc26dc24fefafd99abe5f3de302" candidate_uuid: "3423711e5d5c456abfe9e34a5713238d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128bd3a164cc460bad90618cdd643b70"
I20260926 06:27:36.402099 16211 raft_consensus.cc:3037] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:36.402108 16345 raft_consensus.cc:3037] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:36.402714 16211 raft_consensus.cc:2454] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3423711e5d5c456abfe9e34a5713238d in term 1.
I20260926 06:27:36.402853 16345 raft_consensus.cc:2454] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3423711e5d5c456abfe9e34a5713238d in term 1.
I20260926 06:27:36.402866 16010 leader_election.cc:302] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3423711e5d5c456abfe9e34a5713238d, abe7fcf398a9465c8bbfbe9e9cc323ac; no voters:
I20260926 06:27:36.403035 16416 raft_consensus.cc:2781] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:27:36.403190 16416 raft_consensus.cc:686] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 1 LEADER]: Becoming Leader. State: Replica: 3423711e5d5c456abfe9e34a5713238d, State: Running, Role: LEADER
I20260926 06:27:36.403288 16416 consensus_queue.cc:237] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
I20260926 06:27:36.403991 15931 catalog_manager.cc:5495] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3423711e5d5c456abfe9e34a5713238d (127.2.39.193). New cstate: current_term: 1 leader_uuid: "3423711e5d5c456abfe9e34a5713238d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } health_report { overall_health: UNKNOWN } } }
I20260926 06:27:36.411253 16405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.408854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.2.39.193:45395)
0926 06:27:36.408908 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.408910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.408919 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:27:36.409015 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.409016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.409034 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.409105 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:27:36.409223 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.409742 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.410509 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.410723 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.410789 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.410938 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:27:36.411042 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:27:36.411126 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.411144 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.411163 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 06:27:36.412707 16258 tablet.cc:2369] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:27:36.412789 16211 raft_consensus.cc:1261] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Refusing update from remote peer 3423711e5d5c456abfe9e34a5713238d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:27:36.412803 16345 raft_consensus.cc:1261] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70 [term 1 FOLLOWER]: Refusing update from remote peer 3423711e5d5c456abfe9e34a5713238d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:27:36.413095 16416 consensus_queue.cc:1035] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [LEADER]: Connected to new peer: Peer: permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:27:36.413239 16418 consensus_queue.cc:1035] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:27:36.415508 16421 mvcc.cc:205] Tried to move back new op lower bound from 7333495015064543232 to 7333495015028506624. Current Snapshot: MvccSnapshot[applied={T|T < 7333495015064543232}]
I20260926 06:27:36.416363 16422 mvcc.cc:205] Tried to move back new op lower bound from 7333495015064543232 to 7333495015028506624. Current Snapshot: MvccSnapshot[applied={T|T < 7333495015064543232}]
I20260926 06:27:36.417663 15931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:27:36.417766 15931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:27:36.418467 15931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Sending DeleteTablet for 3 replicas of tablet 44921cc26dc24fefafd99abe5f3de302
I20260926 06:27:36.418763 16057 tablet_service.cc:1514] Processing DeleteTablet for tablet 44921cc26dc24fefafd99abe5f3de302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:36 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:45460
I20260926 06:27:36.418803 16191 tablet_service.cc:1514] Processing DeleteTablet for tablet 44921cc26dc24fefafd99abe5f3de302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:36 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:51698
I20260926 06:27:36.418861 16325 tablet_service.cc:1514] Processing DeleteTablet for tablet 44921cc26dc24fefafd99abe5f3de302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:36 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:54732
I20260926 06:27:36.418946 16431 tablet_replica.cc:330] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: stopping tablet replica
I20260926 06:27:36.419021 16431 raft_consensus.cc:2229] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:27:36.419103 16431 raft_consensus.cc:2258] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:36.419128 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 15994
I20260926 06:27:36.420696 16433 tablet_replica.cc:330] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: stopping tablet replica
I20260926 06:27:36.420802 16433 raft_consensus.cc:2229] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:27:36.420894 16433 raft_consensus.cc:2258] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:36.421056 16432 tablet_replica.cc:330] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: stopping tablet replica
I20260926 06:27:36.421147 16432 raft_consensus.cc:2229] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:27:36.421244 16432 raft_consensus.cc:2258] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:36.421520 16433 ts_tablet_manager.cc:1905] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:27:36.421648 16432 ts_tablet_manager.cc:1905] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:27:36.423316 16432 ts_tablet_manager.cc:1918] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:27:36.423316 16433 ts_tablet_manager.cc:1918] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:27:36.423391 16432 log.cc:1196] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/44921cc26dc24fefafd99abe5f3de302
I20260926 06:27:36.423491 16433 log.cc:1196] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/44921cc26dc24fefafd99abe5f3de302
I20260926 06:27:36.423682 16432 ts_tablet_manager.cc:1939] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Deleting consensus metadata
W20260926 06:27:36.423808 15919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:45395 (error 108)
I20260926 06:27:36.423909 16433 ts_tablet_manager.cc:1939] T 44921cc26dc24fefafd99abe5f3de302 P 128bd3a164cc460bad90618cdd643b70: Deleting consensus metadata
I20260926 06:27:36.424059 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 16128
I20260926 06:27:36.424317 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.424208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45395 (local address 127.0.0.1:45476)
0926 06:27:36.424255 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.424289 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 06:27:36.424423 15919 catalog_manager.cc:4584] TS 3423711e5d5c456abfe9e34a5713238d (127.2.39.193:45395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44921cc26dc24fefafd99abe5f3de302: Network error: Client connection negotiation failed: client connection to 127.2.39.193:45395: connect: Connection refused (error 111)
I20260926 06:27:36.424455 15918 catalog_manager.cc:4841] TS 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687): tablet 44921cc26dc24fefafd99abe5f3de302 (table test-table [id=b1acf0d71d254623b980c28fb2baea94]) successfully deleted
I20260926 06:27:36.429127 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 16262
I20260926 06:27:36.429154 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.429056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:34313 (local address 127.0.0.1:51712)
0926 06:27:36.429100 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.429124 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.194:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 06:27:36.433563 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 15900
I20260926 06:27:36.438711 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:33781
--webserver_interface=127.2.39.254
--webserver_port=43901
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:33781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:36.519217 16435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:36.519421 16435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:36.519446 16435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:36.520558 16435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:27:36.520591 16435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:36.520605 16435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:36.520618 16435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:27:36.520630 16435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:27:36.522076 16435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:33781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:33781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=43901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:36.522301 16435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:36.522490 16435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:36.530928 16444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:36.530928 16442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:36.531162 16435 server_base.cc:1034] running on GCE node
W20260926 06:27:36.530977 16441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:36.531476 16435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:36.531782 16435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:36.532933 16435 hybrid_clock.cc:648] HybridClock initialized: now 1790404056532923 us; error 40 us; skew 500 ppm
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:36.540118 16435 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:27:36.540575 16435 webserver.cc:460] Webserver started at http://127.2.39.254:43901/ using document root <none> and password file <none>
I20260926 06:27:36.540827 16435 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:36.540910 16435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:36.542371 16435 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:36.543027 16451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.543200 16435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:36.543298 16435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "98ad1b2458984a8bb885c7faa50a7bd6"
format_stamp: "Formatted at 2026-09-26 06:27:35 on dist-test-slave-fkbn"
server_key: "19c3b7d45954d53a446c880c4855c190"
server_key_iv: "eeec277243972390afc8867dbbf41d08"
server_key_version: "encryptionkey@0"
I20260926 06:27:36.543711 16435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:36.558928 16435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:36.561198 16435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:36.561353 16435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:36.565194 16435 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:33781
I20260926 06:27:36.565231 16503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:33781 every 8 connection(s)
I20260926 06:27:36.565683 16435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:27:36.567361 16504 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:27:36.567996 16504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Bootstrap starting.
I20260926 06:27:36.573405 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 16435
I20260926 06:27:36.573783 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:45395
--local_ip_for_outbound_sockets=127.2.39.193
--tserver_master_addrs=127.2.39.254:33781
--webserver_port=42053
--webserver_interface=127.2.39.193
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:27:36.576587 16504 log.cc:824] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:36.578720 16504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:27:36.579034 16504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Bootstrap complete.
I20260926 06:27:36.581278 16504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:36.581523 16504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ad1b2458984a8bb885c7faa50a7bd6, State: Initialized, Role: FOLLOWER
I20260926 06:27:36.581732 16504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:36.581822 16504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:27:36.581880 16504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:27:36.581928 16504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:27:36.582937 16504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:36.583060 16504 leader_election.cc:302] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ad1b2458984a8bb885c7faa50a7bd6; no voters:
I20260926 06:27:36.583240 16504 leader_election.cc:288] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:27:36.583281 16508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:27:36.583388 16508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 2 LEADER]: Becoming Leader. State: Replica: 98ad1b2458984a8bb885c7faa50a7bd6, State: Running, Role: LEADER
I20260926 06:27:36.583510 16504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:27:36.583551 16508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:36.584002 16510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ad1b2458984a8bb885c7faa50a7bd6. Latest consensus state: current_term: 2 leader_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } } }
I20260926 06:27:36.584012 16509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } } }
I20260926 06:27:36.584136 16509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:36.584093 16510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:36.584394 16515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:27:36.585247 16515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:27:36.585376 16515 catalog_manager.cc:1259] Loaded cluster ID: 0962f51df9d646a78e36364678855d8d
I20260926 06:27:36.585455 16515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:27:36.585800 16515 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 06:27:36.590453 16515 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 06:27:36.592790 16515 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:27:36.593032 16515 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Loaded TSK: 0
I20260926 06:27:36.593271 16515 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:27:36.657235 16506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:36.657457 16506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:36.657497 16506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:36.658684 16506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:36.658735 16506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:36.658779 16506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:27:36.660149 16506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:45395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=42053
--tserver_master_addrs=127.2.39.254:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:36.660395 16506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:36.660609 16506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:36.665512 16531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:36.665529 16532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:36.665720 16506 server_base.cc:1034] running on GCE node
W20260926 06:27:36.665825 16534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:36.666006 16506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:36.666235 16506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:36.667392 16506 hybrid_clock.cc:648] HybridClock initialized: now 1790404056667377 us; error 30 us; skew 500 ppm
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:36.670079 16506 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:27:36.670545 16506 webserver.cc:460] Webserver started at http://127.2.39.193:42053/ using document root <none> and password file <none>
I20260926 06:27:36.670758 16506 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:36.670831 16506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:36.672207 16506 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:36.672883 16541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.673028 16506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:36.673121 16506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "3423711e5d5c456abfe9e34a5713238d"
format_stamp: "Formatted at 2026-09-26 06:27:35 on dist-test-slave-fkbn"
server_key: "ad09938c5880e0828629241f09f6a5b7"
server_key_iv: "45b833fa231f5409fd1e1bec93366d50"
server_key_version: "encryptionkey@0"
I20260926 06:27:36.673431 16506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:36.690946 16506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:36.693168 16506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:36.693303 16506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:36.693529 16506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:36.694032 16548 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:27:36.695399 16506 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:27:36.695447 16506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:27:36.695479 16506 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:27:36.696103 16506 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:27:36.696146 16506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 06:27:36.696187 16548 tablet_bootstrap.cc:492] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Bootstrap starting.
I20260926 06:27:36.703707 16506 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:45395
I20260926 06:27:36.704000 16655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:45395 every 8 connection(s)
I20260926 06:27:36.704340 16506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:27:36.707396 16548 log.cc:824] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:36.709319 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 16506
I20260926 06:27:36.709730 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:34313
--local_ip_for_outbound_sockets=127.2.39.194
--tserver_master_addrs=127.2.39.254:33781
--webserver_port=43465
--webserver_interface=127.2.39.194
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:27:36.710040 16548 tablet_bootstrap.cc:492] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:27:36.710297 16548 tablet_bootstrap.cc:492] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Bootstrap complete.
I20260926 06:27:36.711432 16548 ts_tablet_manager.cc:1397] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260926 06:27:36.712620 16548 raft_consensus.cc:348] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
I20260926 06:27:36.712954 16548 raft_consensus.cc:729] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3423711e5d5c456abfe9e34a5713238d, State: Initialized, Role: FOLLOWER
I20260926 06:27:36.713095 16548 consensus_queue.cc:260] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } }
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:36.713375 16548 ts_tablet_manager.cc:1428] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:27:36.717393 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.705123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:58681 (local address 127.2.39.254:33781)
0926 06:27:36.705304 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.705313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.705885 (+ 572us) server_negotiation.cc:407] Connection header received
0926 06:27:36.706721 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.706727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.706780 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.706911 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:36.707720 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.709005 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.710582 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.711118 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.711195 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.713995 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:36.713999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:36.713999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:36.714009 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:36.715675 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.716189 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.716192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.716192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.716271 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.716552 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.716553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.716553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.716746 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:36.716809 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.717164 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.717260 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260926 06:27:36.717532 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.705014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:58681)
0926 06:27:36.705714 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.705765 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.706499 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.706970 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.706974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.707341 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:36.707605 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.707611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.709178 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.709180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.710470 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.710473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.711309 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.711310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.711444 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.712064 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:36.712074 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:36.713887 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:36.715750 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.715757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.715759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.716101 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.716335 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.716336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.716336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.716478 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.716812 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:36.716818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:36.716994 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.717109 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.717293 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":88,"threads_started":1}
I20260926 06:27:36.718420 16656 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33781
I20260926 06:27:36.718525 16656 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:36.718719 16656 heartbeater.cc:511] Master 127.2.39.254:33781 requested a full tablet report, sending...
I20260926 06:27:36.719452 16468 ts_manager.cc:194] Registered new tserver with Master: 3423711e5d5c456abfe9e34a5713238d (127.2.39.193:45395)
I20260926 06:27:36.720544 16468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:58681
I20260926 06:27:36.721376 16656 heartbeater.cc:503] Master 127.2.39.254:33781 was elected leader, sending a full tablet report...
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:27:36.728786 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.719998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45395 (local address 127.0.0.1:45480)
0926 06:27:36.720160 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.720201 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.720280 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.721027 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.721033 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.721489 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:36.721554 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.721557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.722964 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.722965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.723522 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.723523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.724161 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.724162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.724211 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.724660 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:36.724664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:36.725888 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:36.727756 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.727761 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.727762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.727963 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.728177 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.728177 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.728178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.728227 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.728536 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:36.728541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:36.728580 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.728643 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.728723 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260926 06:27:36.728868 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.720179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.2.39.193:45395)
0926 06:27:36.720836 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.720839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.720867 (+ 28us) server_negotiation.cc:407] Connection header received
0926 06:27:36.720899 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.720901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.720926 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.721009 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:36.721704 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.722866 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.723605 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.724081 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.724164 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.725971 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:36.725978 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:36.725978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:36.725985 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:36.727676 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.728038 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.728041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.728041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.728104 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.728282 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.728283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.728283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.728468 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:36.728557 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.728677 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.728771 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":59,"threads_started":1}
I20260926 06:27:36.729250 16590 tablet_service.cc:1514] Processing DeleteTablet for tablet 44921cc26dc24fefafd99abe5f3de302 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:36 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:45480
I20260926 06:27:36.729880 16665 tablet_replica.cc:330] stopping tablet replica
I20260926 06:27:36.729995 16665 raft_consensus.cc:2229] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:27:36.730079 16665 raft_consensus.cc:2258] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:36.730350 16665 ts_tablet_manager.cc:1905] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:27:36.732472 16665 ts_tablet_manager.cc:1918] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:27:36.732563 16665 log.cc:1196] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/44921cc26dc24fefafd99abe5f3de302
I20260926 06:27:36.732779 16665 ts_tablet_manager.cc:1939] T 44921cc26dc24fefafd99abe5f3de302 P 3423711e5d5c456abfe9e34a5713238d: Deleting consensus metadata
I20260926 06:27:36.733229 16455 catalog_manager.cc:4841] TS 3423711e5d5c456abfe9e34a5713238d (127.2.39.193:45395): tablet 44921cc26dc24fefafd99abe5f3de302 (table test-table [id=b1acf0d71d254623b980c28fb2baea94]) successfully deleted
W20260926 06:27:36.799422 16661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:36.799618 16661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:36.799641 16661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:36.800679 16661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:36.800719 16661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:36.800746 16661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:27:36.802140 16661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:34313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=43465
--tserver_master_addrs=127.2.39.254:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:36.802330 16661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:36.802505 16661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:36.807586 16671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:36.807612 16673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:36.807785 16661 server_base.cc:1034] running on GCE node
W20260926 06:27:36.807590 16670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:36.808050 16661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:36.808257 16661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:36.809522 16661 hybrid_clock.cc:648] HybridClock initialized: now 1790404056809508 us; error 28 us; skew 500 ppm
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:36.811821 16661 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:27:36.812301 16661 webserver.cc:460] Webserver started at http://127.2.39.194:43465/ using document root <none> and password file <none>
I20260926 06:27:36.812487 16661 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:36.812541 16661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:36.813854 16661 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:27:36.814455 16680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.814611 16661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:27:36.814704 16661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac"
format_stamp: "Formatted at 2026-09-26 06:27:36 on dist-test-slave-fkbn"
server_key: "1000294db1a88ba7b89448a73db137fc"
server_key_iv: "c13fb4c662f902b0748eb9abab89bbcc"
server_key_version: "encryptionkey@0"
I20260926 06:27:36.815054 16661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:36.830552 16661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:36.832312 16661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:36.832475 16661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:36.832695 16661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:36.833127 16687 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 06:27:36.834595 16687 ts_tablet_manager.cc:1852] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 06:27:36.834851 16687 ts_tablet_manager.cc:1905] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:27:36.837018 16687 ts_tablet_manager.cc:1918] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:27:36.837188 16687 ts_tablet_manager.cc:1939] T 44921cc26dc24fefafd99abe5f3de302 P abe7fcf398a9465c8bbfbe9e9cc323ac: Deleting consensus metadata
I20260926 06:27:36.837409 16661 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 06:27:36.837491 16661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 06:27:36.837553 16661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:36.837592 16661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.843343 16661 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:34313
I20260926 06:27:36.843394 16794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:34313 every 8 connection(s)
I20260926 06:27:36.843679 16661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:27:36.847386 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 16661
I20260926 06:27:36.847690 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:44687
--local_ip_for_outbound_sockets=127.2.39.195
--tserver_master_addrs=127.2.39.254:33781
--webserver_port=37261
--webserver_interface=127.2.39.195
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:36.853266 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.844329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:44219 (local address 127.2.39.254:33781)
0926 06:27:36.844375 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.844378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.844567 (+ 189us) server_negotiation.cc:407] Connection header received
0926 06:27:36.845052 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.845053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.845062 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.845107 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:36.845760 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.846273 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.847319 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.847549 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.847612 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.850249 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:36.850251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:36.850252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:36.850261 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:36.851887 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.852263 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.852265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.852266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.852330 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.852565 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.852566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.852566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.852720 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:36.852796 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.853024 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.853127 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:27:36.853276 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.844282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:44219)
0926 06:27:36.844442 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.844483 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.844942 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.845181 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.845184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.845423 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:36.845679 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.845684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.846335 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.846335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.847241 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.847244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.847634 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.847635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.847783 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.848293 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:36.848303 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:36.850152 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:36.851965 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.851976 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.851978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.852192 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.852402 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.852402 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.852402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.852507 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.852784 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:36.852788 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:36.852898 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.852989 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.853111 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260926 06:27:36.853823 16795 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33781
I20260926 06:27:36.853911 16795 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:36.854096 16795 heartbeater.cc:511] Master 127.2.39.254:33781 requested a full tablet report, sending...
I20260926 06:27:36.854463 16468 ts_manager.cc:194] Registered new tserver with Master: abe7fcf398a9465c8bbfbe9e9cc323ac (127.2.39.194:34313)
I20260926 06:27:36.854826 16468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:44219
W20260926 06:27:36.929286 16798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:36.929514 16798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:36.929551 16798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:36.930706 16798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:36.930776 16798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:36.930821 16798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:27:36.932243 16798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:44687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=37261
--tserver_master_addrs=127.2.39.254:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:36.932504 16798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:36.932725 16798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:27:36.937732 16798 server_base.cc:1034] running on GCE node
W20260926 06:27:36.937717 16804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:36.937707 16803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:36.937741 16806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:36.938082 16798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:36.938293 16798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:36.939422 16798 hybrid_clock.cc:648] HybridClock initialized: now 1790404056939404 us; error 33 us; skew 500 ppm
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:36.941727 16798 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:27:36.942142 16798 webserver.cc:460] Webserver started at http://127.2.39.195:37261/ using document root <none> and password file <none>
I20260926 06:27:36.942317 16798 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:36.942365 16798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:36.943573 16798 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:36.944082 16813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.944231 16798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:36.944316 16798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "128bd3a164cc460bad90618cdd643b70"
format_stamp: "Formatted at 2026-09-26 06:27:36 on dist-test-slave-fkbn"
server_key: "5038c868c075fab04211736d75f44adc"
server_key_iv: "73b5a9522453b26ffd25b26ded4b301c"
server_key_version: "encryptionkey@0"
I20260926 06:27:36.944617 16798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:36.959782 16798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:36.961424 16798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:36.961527 16798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:36.961763 16798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:36.962004 16798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:36.962034 16798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.962054 16798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:36.962066 16798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:36.967512 16798 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:44687
I20260926 06:27:36.967581 16926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:44687 every 8 connection(s)
I20260926 06:27:36.967904 16798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:27:36.973104 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 16798
Sep 26 06:27:36 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404056, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:36.977229 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.968611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:54545 (local address 127.2.39.254:33781)
0926 06:27:36.968670 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.968672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.968866 (+ 194us) server_negotiation.cc:407] Connection header received
0926 06:27:36.969452 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.969454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.969464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.969527 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:36.970409 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.970869 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.971766 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.971936 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.971987 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.974353 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:36.974355 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:36.974355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:36.974362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:36.975804 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.976247 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.976248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.976248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.976301 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.976577 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.976579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.976579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.976747 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:36.976816 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.977050 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.977128 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 06:27:36.977367 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.968588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:54545)
0926 06:27:36.968745 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.968787 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.969343 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.969604 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.969607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.970154 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:36.970335 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.970339 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.970940 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.970940 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.971699 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.971701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.971998 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.971999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.972100 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.972654 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:36.972668 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:36.974258 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:36.975879 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.975885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.975888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.976156 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.976380 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:36.976381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:36.976382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:36.976511 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:36.976804 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:36.976809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:36.976924 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.977004 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.977164 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260926 06:27:36.977808 16927 heartbeater.cc:347] Connected to a master server at 127.2.39.254:33781
I20260926 06:27:36.977909 16927 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:36.978101 16927 heartbeater.cc:511] Master 127.2.39.254:33781 requested a full tablet report, sending...
I20260926 06:27:36.978518 16468 ts_manager.cc:194] Registered new tserver with Master: 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687)
I20260926 06:27:36.978860 16468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:54545
I20260926 06:27:36.980127 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.973418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58058 (local address 127.2.39.254:33781)
0926 06:27:36.973551 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.973553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.973593 (+ 40us) server_negotiation.cc:407] Connection header received
0926 06:27:36.973665 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.973666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.973677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.973757 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:36.974324 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.974949 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.975689 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.975900 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.975995 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.977292 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:36.977293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:36.977293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:36.977300 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:36.978833 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.979562 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.979563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.979563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.979613 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:36.979785 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:36.979785 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:36.979786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:36.979869 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:36.979986 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.980013 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.980071 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260926 06:27:36.980800 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:27:36.983455 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.981128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58062 (local address 127.2.39.254:33781)
0926 06:27:36.981175 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.981177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.981239 (+ 62us) server_negotiation.cc:407] Connection header received
0926 06:27:36.981320 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.981321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.981355 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.981432 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:27:36.981531 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.981996 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.982730 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.982929 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.983006 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.983166 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:27:36.983258 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:27:36.983345 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.983363 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.983384 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:27:36.983726 16468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:27:36.985337 16468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:27:36.987401 16590 tablet_service.cc:1467] Processing CreateTablet for tablet 802704a03047494e9bce6f8a74145726 (DEFAULT_TABLE table=test-table [id=17b82f91827c47c0bf0976b4c1de58df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:36.987608 16590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802704a03047494e9bce6f8a74145726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:36.988933 16548 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Bootstrap starting.
I20260926 06:27:36.989285 16548 tablet_bootstrap.cc:654] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:36.989917 16548 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: No bootstrap required, opened a new log
I20260926 06:27:36.989984 16548 ts_tablet_manager.cc:1397] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:36.990337 16548 raft_consensus.cc:348] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:36.990401 16548 raft_consensus.cc:374] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:36.990443 16548 raft_consensus.cc:729] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3423711e5d5c456abfe9e34a5713238d, State: Initialized, Role: FOLLOWER
I20260926 06:27:36.990489 16548 consensus_queue.cc:260] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:36.990569 16548 ts_tablet_manager.cc:1428] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 06:27:36.990945 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.987052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:44687 (local address 127.0.0.1:54742)
0926 06:27:36.987145 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.987155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.987201 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.987619 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.987633 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.987634 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:36.987683 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.987684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.988739 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.988741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.989715 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.989718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.990662 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.990663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.990754 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.990776 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.990832 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.990880 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 06:27:36.991194 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.987138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.2.39.195:44687)
0926 06:27:36.987458 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.987464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.987473 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:27:36.987510 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.987521 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.987537 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.987613 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:36.987755 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.988637 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.989802 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.990504 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.990573 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.990630 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.991068 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.991116 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":90,"threads_started":1}
I20260926 06:27:36.991820 16861 tablet_service.cc:1467] Processing CreateTablet for tablet 802704a03047494e9bce6f8a74145726 (DEFAULT_TABLE table=test-table [id=17b82f91827c47c0bf0976b4c1de58df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:36.992177 16861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802704a03047494e9bce6f8a74145726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:36.992954 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.987395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.2.39.194:34313)
0926 06:27:36.989013 (+ 1618us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:36.989017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:36.989028 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:27:36.989069 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:36.989071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:36.989091 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:36.989183 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:36.989385 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.990570 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.991875 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.992647 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.992700 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.992734 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:36.992868 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 06:27:36.992896 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1582,"thread_start_us":90,"threads_started":1}
I20260926 06:27:36.992956 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:36.987113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:34313 (local address 127.0.0.1:51720)
0926 06:27:36.987282 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:36.987290 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:36.987345 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:36.989254 (+ 1909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:36.989255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:36.989257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:36.989300 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.989301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.990688 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.990689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.991792 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:36.991795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:36.992722 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:36.992724 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:36.992790 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:36.992794 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:27:36.992855 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:27:36.992891 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":114,"threads_started":1}
I20260926 06:27:36.993449 16729 tablet_service.cc:1467] Processing CreateTablet for tablet 802704a03047494e9bce6f8a74145726 (DEFAULT_TABLE table=test-table [id=17b82f91827c47c0bf0976b4c1de58df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:36.993749 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802704a03047494e9bce6f8a74145726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:36.995064 16937 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Bootstrap starting.
I20260926 06:27:36.995330 16687 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Bootstrap starting.
I20260926 06:27:36.995746 16937 tablet_bootstrap.cc:654] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:36.995893 16687 tablet_bootstrap.cc:654] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:36.996387 16937 log.cc:824] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:36.996439 16687 log.cc:824] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:36.997126 16937 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: No bootstrap required, opened a new log
I20260926 06:27:36.997193 16937 ts_tablet_manager.cc:1397] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:27:36.997265 16687 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: No bootstrap required, opened a new log
I20260926 06:27:36.997321 16687 ts_tablet_manager.cc:1397] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:27:36.999146 16937 raft_consensus.cc:348] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:36.999212 16687 raft_consensus.cc:348] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:36.999327 16937 raft_consensus.cc:374] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:36.999330 16687 raft_consensus.cc:374] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:36.999363 16687 raft_consensus.cc:729] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe7fcf398a9465c8bbfbe9e9cc323ac, State: Initialized, Role: FOLLOWER
I20260926 06:27:36.999363 16937 raft_consensus.cc:729] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128bd3a164cc460bad90618cdd643b70, State: Initialized, Role: FOLLOWER
I20260926 06:27:36.999476 16687 consensus_queue.cc:260] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:36.999485 16937 consensus_queue.cc:260] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:36.999750 16687 ts_tablet_manager.cc:1428] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:27:36.999750 16937 ts_tablet_manager.cc:1428] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:27:36.999771 16927 heartbeater.cc:503] Master 127.2.39.254:33781 was elected leader, sending a full tablet report...
I20260926 06:27:36.999857 16795 heartbeater.cc:503] Master 127.2.39.254:33781 was elected leader, sending a full tablet report...
I20260926 06:27:37.031987 16940 raft_consensus.cc:482] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:27:37.032160 16940 raft_consensus.cc:504] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:37.032554 16940 leader_election.cc:288] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687), 3423711e5d5c456abfe9e34a5713238d (127.2.39.193:45395)
I20260926 06:27:37.036459 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.032787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:45273 (local address 127.2.39.195:44687)
0926 06:27:37.032842 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:37.032844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:37.033047 (+ 203us) server_negotiation.cc:407] Connection header received
0926 06:27:37.033078 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:37.033079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:37.033093 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:37.033134 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:37.033276 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:37.033997 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:37.035545 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:37.036099 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:37.036180 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:37.036191 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:37.036350 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 06:27:37.036391 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:27:37.036459 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.032812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:36787 (local address 127.2.39.193:45395)
0926 06:27:37.032862 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:37.032864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:37.032874 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:27:37.032902 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:37.032903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:37.032927 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:37.033000 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:37.033195 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:37.034017 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:37.035550 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:37.036149 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:37.036214 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:37.036291 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:37.036348 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 06:27:37.036387 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:27:37.036486 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.032710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45395 (local address 127.2.39.194:36787)
0926 06:27:37.032761 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.032774 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:37.032820 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:37.033033 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:37.033034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:37.033036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:37.033110 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:37.033112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:37.034090 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:37.034091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:37.035444 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:37.035446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:37.036217 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:37.036218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:37.036287 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:37.036289 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:27:37.036338 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:27:37.036386 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":100}
I20260926 06:27:37.036492 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.032714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:44687 (local address 127.2.39.194:45273)
0926 06:27:37.032955 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.032962 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:37.033033 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:37.033150 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:37.033151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:37.033152 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:37.033211 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:37.033212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:37.034090 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:37.034091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:37.035444 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:37.035447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:37.036196 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:37.036197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:37.036273 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:37.036277 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:27:37.036336 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 06:27:37.036386 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":125,"thread_start_us":139,"threads_started":1}
I20260926 06:27:37.036790 16610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802704a03047494e9bce6f8a74145726" candidate_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3423711e5d5c456abfe9e34a5713238d" is_pre_election: true
I20260926 06:27:37.036934 16610 raft_consensus.cc:2454] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abe7fcf398a9465c8bbfbe9e9cc323ac in term 0.
I20260926 06:27:37.036960 16881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802704a03047494e9bce6f8a74145726" candidate_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128bd3a164cc460bad90618cdd643b70" is_pre_election: true
I20260926 06:27:37.037137 16881 raft_consensus.cc:2454] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abe7fcf398a9465c8bbfbe9e9cc323ac in term 0.
I20260926 06:27:37.037191 16684 leader_election.cc:302] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3423711e5d5c456abfe9e34a5713238d, abe7fcf398a9465c8bbfbe9e9cc323ac; no voters:
I20260926 06:27:37.037361 16940 raft_consensus.cc:2781] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:27:37.037417 16940 raft_consensus.cc:482] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:27:37.037446 16940 raft_consensus.cc:3037] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:37.038125 16940 raft_consensus.cc:504] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:37.038264 16940 leader_election.cc:288] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [CANDIDATE]: Term 1 election: Requested vote from peers 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687), 3423711e5d5c456abfe9e34a5713238d (127.2.39.193:45395)
I20260926 06:27:37.038393 16881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802704a03047494e9bce6f8a74145726" candidate_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "128bd3a164cc460bad90618cdd643b70"
I20260926 06:27:37.038404 16610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802704a03047494e9bce6f8a74145726" candidate_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3423711e5d5c456abfe9e34a5713238d"
I20260926 06:27:37.038479 16610 raft_consensus.cc:3037] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:37.038484 16881 raft_consensus.cc:3037] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:37.039146 16610 raft_consensus.cc:2454] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abe7fcf398a9465c8bbfbe9e9cc323ac in term 1.
I20260926 06:27:37.039217 16881 raft_consensus.cc:2454] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abe7fcf398a9465c8bbfbe9e9cc323ac in term 1.
I20260926 06:27:37.039296 16684 leader_election.cc:302] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3423711e5d5c456abfe9e34a5713238d, abe7fcf398a9465c8bbfbe9e9cc323ac; no voters:
I20260926 06:27:37.039403 16940 raft_consensus.cc:2781] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:27:37.039573 16940 raft_consensus.cc:686] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 LEADER]: Becoming Leader. State: Replica: abe7fcf398a9465c8bbfbe9e9cc323ac, State: Running, Role: LEADER
I20260926 06:27:37.039662 16940 consensus_queue.cc:237] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:37.040405 16468 catalog_manager.cc:5495] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac reported cstate change: term changed from 0 to 1, leader changed from <none> to abe7fcf398a9465c8bbfbe9e9cc323ac (127.2.39.194). New cstate: current_term: 1 leader_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } health_report { overall_health: UNKNOWN } } }
I20260926 06:27:37.048309 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.045936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.2.39.194:34313)
0926 06:27:37.045989 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:37.045991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:37.046000 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:27:37.046054 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:37.046055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:37.046071 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:37.046148 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:27:37.046257 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:37.046757 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:37.047505 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:37.047744 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:37.047821 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:37.047988 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:27:37.048085 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:27:37.048163 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:37.048203 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 06:27:37.048221 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:27:37.049877 16610 raft_consensus.cc:1261] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Refusing update from remote peer abe7fcf398a9465c8bbfbe9e9cc323ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:27:37.049897 16881 raft_consensus.cc:1261] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 1 FOLLOWER]: Refusing update from remote peer abe7fcf398a9465c8bbfbe9e9cc323ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:27:37.050195 16943 consensus_queue.cc:1035] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:27:37.050328 16940 consensus_queue.cc:1035] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:27:37.052600 16946 mvcc.cc:205] Tried to move back new op lower bound from 7333495017673875456 to 7333495017635168256. Current Snapshot: MvccSnapshot[applied={T|T < 7333495017673875456 or (T in {7333495017673875456})}]
I20260926 06:27:37.054391 16468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:27:37.054472 16468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:27:37.055115 16468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Sending DeleteTablet for 3 replicas of tablet 802704a03047494e9bce6f8a74145726
I20260926 06:27:37.055408 16590 tablet_service.cc:1514] Processing DeleteTablet for tablet 802704a03047494e9bce6f8a74145726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:37 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:45480
I20260926 06:27:37.055394 16729 tablet_service.cc:1514] Processing DeleteTablet for tablet 802704a03047494e9bce6f8a74145726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:37 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:51720
I20260926 06:27:37.055466 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 16506
I20260926 06:27:37.055503 16665 tablet_replica.cc:330] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: stopping tablet replica
I20260926 06:27:37.055474 16861 tablet_service.cc:1514] Processing DeleteTablet for tablet 802704a03047494e9bce6f8a74145726 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:37 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:54742
I20260926 06:27:37.055552 16665 raft_consensus.cc:2229] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:27:37.055604 16665 raft_consensus.cc:2258] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:37.055676 16957 tablet_replica.cc:330] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: stopping tablet replica
I20260926 06:27:37.055757 16957 raft_consensus.cc:2229] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:27:37.055843 16957 raft_consensus.cc:2258] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:37.056246 16957 ts_tablet_manager.cc:1905] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:27:37.058156 16957 ts_tablet_manager.cc:1918] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:27:37.058226 16957 log.cc:1196] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/802704a03047494e9bce6f8a74145726
I20260926 06:27:37.058518 16957 ts_tablet_manager.cc:1939] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Deleting consensus metadata
I20260926 06:27:37.058925 16454 catalog_manager.cc:4841] TS 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687): tablet 802704a03047494e9bce6f8a74145726 (table test-table [id=17b82f91827c47c0bf0976b4c1de58df]) successfully deleted
I20260926 06:27:37.061686 16956 tablet_replica.cc:330] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: stopping tablet replica
I20260926 06:27:37.061790 16956 raft_consensus.cc:2229] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:27:37.061882 16956 raft_consensus.cc:2258] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 06:27:37.062346 16455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:45395 (error 108)
I20260926 06:27:37.062776 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 16661
I20260926 06:27:37.062245 16956 ts_tablet_manager.cc:1905] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:27:37.062899 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.062767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45395 (local address 127.0.0.1:45496)
0926 06:27:37.062828 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.062868 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:45395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 06:27:37.064687 16455 catalog_manager.cc:4584] TS 3423711e5d5c456abfe9e34a5713238d (127.2.39.193:45395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 802704a03047494e9bce6f8a74145726: Network error: Client connection negotiation failed: client connection to 127.2.39.193:45395: connect: Connection refused (error 111)
I20260926 06:27:37.068310 16933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.068212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:34313 (local address 127.0.0.1:51742)
0926 06:27:37.068255 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.068279 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.194:34313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 06:27:37.068305 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 16798
I20260926 06:27:37.072701 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 16435
I20260926 06:27:37.078228 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:33781
--webserver_interface=127.2.39.254
--webserver_port=43901
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:33781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:37.157819 16959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:37.157999 16959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:37.158020 16959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:37.159050 16959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:27:37.159083 16959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:37.159097 16959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:37.159109 16959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:27:37.159121 16959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:27:37.160503 16959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:33781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:33781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=43901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:37.160677 16959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:37.160835 16959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:37.165697 16965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:37.165715 16967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:37.165745 16959 server_base.cc:1034] running on GCE node
W20260926 06:27:37.165706 16964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:37.166149 16959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:37.166375 16959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:37.167495 16959 hybrid_clock.cc:648] HybridClock initialized: now 1790404057167482 us; error 29 us; skew 500 ppm
Sep 26 06:27:37 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:37.169867 16959 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:27:37.170310 16959 webserver.cc:460] Webserver started at http://127.2.39.254:43901/ using document root <none> and password file <none>
I20260926 06:27:37.170545 16959 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:37.170610 16959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:37.171846 16959 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:37.172498 16974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:37.172703 16959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:37.172788 16959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "98ad1b2458984a8bb885c7faa50a7bd6"
format_stamp: "Formatted at 2026-09-26 06:27:35 on dist-test-slave-fkbn"
server_key: "19c3b7d45954d53a446c880c4855c190"
server_key_iv: "eeec277243972390afc8867dbbf41d08"
server_key_version: "encryptionkey@0"
I20260926 06:27:37.173117 16959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:37.192447 16959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:37.194275 16959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:37.194429 16959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:37.198315 16959 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:33781
I20260926 06:27:37.198354 17026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:33781 every 8 connection(s)
I20260926 06:27:37.198738 16959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:27:37.200214 17027 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:27:37.200780 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Bootstrap starting.
I20260926 06:27:37.202621 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 16959
I20260926 06:27:37.202883 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:45395
--local_ip_for_outbound_sockets=127.2.39.193
--tserver_master_addrs=127.2.39.254:33781
--webserver_port=42053
--webserver_interface=127.2.39.193
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:27:37.210062 17027 log.cc:824] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:37.212273 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:27:37.212581 17027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6: Bootstrap complete.
I20260926 06:27:37.214843 17027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:37.215104 17027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ad1b2458984a8bb885c7faa50a7bd6, State: Initialized, Role: FOLLOWER
I20260926 06:27:37.215291 17027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:37.215379 17027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:27:37.215431 17027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:27:37.215477 17027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 2 FOLLOWER]: Advancing to term 3
I20260926 06:27:37.216483 17027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:37.216595 17027 leader_election.cc:302] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ad1b2458984a8bb885c7faa50a7bd6; no voters:
I20260926 06:27:37.216768 17027 leader_election.cc:288] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 06:27:37.216826 17031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 06:27:37.216964 17031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [term 3 LEADER]: Becoming Leader. State: Replica: 98ad1b2458984a8bb885c7faa50a7bd6, State: Running, Role: LEADER
I20260926 06:27:37.217024 17027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:27:37.217070 17031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } }
I20260926 06:27:37.217342 17031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ad1b2458984a8bb885c7faa50a7bd6. Latest consensus state: current_term: 3 leader_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } } }
I20260926 06:27:37.217532 17031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:37.217344 17032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ad1b2458984a8bb885c7faa50a7bd6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 33781 } } }
I20260926 06:27:37.217784 17041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:27:37.217873 17032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ad1b2458984a8bb885c7faa50a7bd6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:37.218643 17041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:27:37.218739 17041 catalog_manager.cc:1259] Loaded cluster ID: 0962f51df9d646a78e36364678855d8d
I20260926 06:27:37.218767 17041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 06:27:37.219003 17041 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f47227e59ad google::LogMessage::Fail() at ??:0
@ 0x7f47227e888a google::LogMessage::SendToLog() at ??:0
@ 0x7f47227e54de google::LogMessage::Flush() at ??:0
@ 0x7f47227e6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f472a7060f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f472a70f8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f47246b30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f47246a8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f472550c6db start_thread at ??:0
@ 0x7f47212e361f clone at ??:0
W20260926 06:27:37.295650 17029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:37.295862 17029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:37.295899 17029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:37.297024 17029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:37.297083 17029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:37.297132 17029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:27:37.298652 17029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:45395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=42053
--tserver_master_addrs=127.2.39.254:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:37.298905 17029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:37.299129 17029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:37.304531 17063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:37.304693 17029 server_base.cc:1034] running on GCE node
W20260926 06:27:37.304751 17064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:37.304822 17066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:37.305019 17029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:37.305280 17029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:37.306465 17029 hybrid_clock.cc:648] HybridClock initialized: now 1790404057306446 us; error 33 us; skew 500 ppm
Sep 26 06:27:37 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:37.309393 17029 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:27:37.309857 17029 webserver.cc:460] Webserver started at http://127.2.39.193:42053/ using document root <none> and password file <none>
I20260926 06:27:37.310109 17029 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:37.310154 17029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:37.311429 17029 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:37.312093 17073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:37.312291 17029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:27:37.312381 17029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "3423711e5d5c456abfe9e34a5713238d"
format_stamp: "Formatted at 2026-09-26 06:27:35 on dist-test-slave-fkbn"
server_key: "ad09938c5880e0828629241f09f6a5b7"
server_key_iv: "45b833fa231f5409fd1e1bec93366d50"
server_key_version: "encryptionkey@0"
I20260926 06:27:37.312716 17029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:37.333299 17029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:37.335428 17029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:37.335592 17029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:37.335817 17029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:37.336251 17080 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:27:37.338306 17029 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:27:37.338359 17029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:27:37.338402 17029 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:27:37.338973 17029 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:27:37.339030 17029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 06:27:37.339099 17080 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Bootstrap starting.
I20260926 06:27:37.346338 17029 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:45395
I20260926 06:27:37.346410 17187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:45395 every 8 connection(s)
I20260926 06:27:37.346732 17029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:27:37.347949 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 17029
I20260926 06:27:37.348250 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:34313
--local_ip_for_outbound_sockets=127.2.39.194
--tserver_master_addrs=127.2.39.254:33781
--webserver_port=43465
--webserver_interface=127.2.39.194
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:27:37.350411 17080 log.cc:824] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:37.353137 17080 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:27:37.353449 17080 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Bootstrap complete.
I20260926 06:27:37.354703 17080 ts_tablet_manager.cc:1397] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.013s
I20260926 06:27:37.356124 17080 raft_consensus.cc:348] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:37.356442 17080 raft_consensus.cc:729] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3423711e5d5c456abfe9e34a5713238d, State: Initialized, Role: FOLLOWER
I20260926 06:27:37.356595 17080 consensus_queue.cc:260] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:37.356848 17080 ts_tablet_manager.cc:1428] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
W20260926 06:27:37.410080 17190 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:27:37.347428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:48005)
0926 06:27:37.347598 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.347631 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:37.348602 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:37.409886 (+ 61284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
W20260926 06:27:37.410328 17188 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:33781 (0 consecutive failures): Network error: Failed to ping master at 127.2.39.254:33781: Client connection negotiation failed: client connection to 127.2.39.254:33781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:27:37.410669 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.410563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:57379)
0926 06:27:37.410610 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.410650 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 06:27:37.411018 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.410937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:47483)
0926 06:27:37.410983 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.411004 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 06:27:37.411130 17188 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:27:37.430802 17191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:37.431010 17191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:37.431087 17191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:37.432219 17191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:37.432279 17191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:37.432324 17191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:27:37.433842 17191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:34313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=43465
--tserver_master_addrs=127.2.39.254:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:37.434094 17191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:37.434299 17191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:37.439077 17198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:37.439086 17201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:37.439121 17191 server_base.cc:1034] running on GCE node
W20260926 06:27:37.439105 17199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:37.439472 17191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:37.439666 17191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:37.440793 17191 hybrid_clock.cc:648] HybridClock initialized: now 1790404057440783 us; error 25 us; skew 500 ppm
Sep 26 06:27:37 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:37.443089 17191 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:27:37.443483 17191 webserver.cc:460] Webserver started at http://127.2.39.194:43465/ using document root <none> and password file <none>
I20260926 06:27:37.443665 17191 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:37.443706 17191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:37.445297 17191 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:27:37.445906 17208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:37.446091 17191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:37.446175 17191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac"
format_stamp: "Formatted at 2026-09-26 06:27:36 on dist-test-slave-fkbn"
server_key: "1000294db1a88ba7b89448a73db137fc"
server_key_iv: "c13fb4c662f902b0748eb9abab89bbcc"
server_key_version: "encryptionkey@0"
I20260926 06:27:37.446465 17191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:37.466607 17191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:37.468434 17191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:37.468618 17191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:37.468854 17191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:37.469275 17215 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:27:37.470728 17191 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:27:37.470781 17191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:27:37.470822 17191 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:27:37.471405 17191 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:27:37.471457 17191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 06:27:37.471503 17215 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Bootstrap starting.
I20260926 06:27:37.472846 17215 log.cc:824] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:37.475039 17215 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:27:37.475399 17215 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Bootstrap complete.
I20260926 06:27:37.475546 17215 ts_tablet_manager.cc:1397] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 06:27:37.476783 17215 raft_consensus.cc:348] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:37.477033 17215 raft_consensus.cc:729] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abe7fcf398a9465c8bbfbe9e9cc323ac, State: Initialized, Role: FOLLOWER
I20260926 06:27:37.477195 17215 consensus_queue.cc:260] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:37.477453 17215 ts_tablet_manager.cc:1428] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260926 06:27:37.478610 17191 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:34313
I20260926 06:27:37.478700 17324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:34313 every 8 connection(s)
I20260926 06:27:37.478940 17191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:27:37.479786 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.479498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:58031)
0926 06:27:37.479629 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.479687 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
W20260926 06:27:37.479952 17325 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:33781 (0 consecutive failures): Network error: Failed to ping master at 127.2.39.254:33781: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
I20260926 06:27:37.480202 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.480117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:35755)
0926 06:27:37.480158 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.480181 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 06:27:37.480543 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.480437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:48575)
0926 06:27:37.480482 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.480509 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 06:27:37.480636 17325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:27:37.483188 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 17191
I20260926 06:27:37.483409 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:44687
--local_ip_for_outbound_sockets=127.2.39.195
--tserver_master_addrs=127.2.39.254:33781
--webserver_port=37261
--webserver_interface=127.2.39.195
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:37.562569 17328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:37.562803 17328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:37.562844 17328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:37.563982 17328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:37.564047 17328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:37.564090 17328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:27:37.565590 17328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:44687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=37261
--tserver_master_addrs=127.2.39.254:33781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:37.565850 17328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:37.566079 17328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:37.571211 17334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:37.571211 17336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:37.571238 17333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:37.571389 17328 server_base.cc:1034] running on GCE node
I20260926 06:27:37.571656 17328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:37.571874 17328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:37.573027 17328 hybrid_clock.cc:648] HybridClock initialized: now 1790404057573016 us; error 37 us; skew 500 ppm
Sep 26 06:27:37 dist-test-slave-fkbn krb5kdc[15887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:37.575343 17328 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:27:37.575749 17328 webserver.cc:460] Webserver started at http://127.2.39.195:37261/ using document root <none> and password file <none>
I20260926 06:27:37.575968 17328 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:37.576032 17328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:37.577302 17328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:27:37.577872 17343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:37.578019 17328 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:37.578110 17328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "128bd3a164cc460bad90618cdd643b70"
format_stamp: "Formatted at 2026-09-26 06:27:36 on dist-test-slave-fkbn"
server_key: "5038c868c075fab04211736d75f44adc"
server_key_iv: "73b5a9522453b26ffd25b26ded4b301c"
server_key_version: "encryptionkey@0"
I20260926 06:27:37.578438 17328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:37.600296 17328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:37.602926 17328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:37.603084 17328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:37.603317 17328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:37.603674 17328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:37.603724 17328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:37.603762 17328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:37.603789 17328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:37.611500 17328 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:44687
I20260926 06:27:37.611598 17456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:44687 every 8 connection(s)
I20260926 06:27:37.611925 17328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:27:37.613035 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.612543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:37949)
0926 06:27:37.612736 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.612837 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":73,"threads_started":1}
W20260926 06:27:37.613552 17457 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:33781 (0 consecutive failures): Network error: Failed to ping master at 127.2.39.254:33781: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
I20260926 06:27:37.613896 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.613775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:33473)
0926 06:27:37.613831 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.613865 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 06:27:37.614245 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:37.614168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:44247)
0926 06:27:37.614204 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:37.614225 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 06:27:37.614346 17457 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:27:37.618295 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 17328
I20260926 06:27:38.411751 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:38.411540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:38705)
0926 06:27:38.411659 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:38.411690 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260926 06:27:38.481078 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:38.480902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:41249)
0926 06:27:38.481010 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:38.481040 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260926 06:27:38.614946 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:38.614740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:35123)
0926 06:27:38.614866 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:38.614900 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":53,"threads_started":1}
I20260926 06:27:38.780783 17464 raft_consensus.cc:482] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:27:38.780936 17464 raft_consensus.cc:504] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:38.781289 17464 leader_election.cc:288] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687), abe7fcf398a9465c8bbfbe9e9cc323ac (127.2.39.194:34313)
Sep 26 06:27:38 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.194@KRBTEST.COM
Sep 26 06:27:38 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.195@KRBTEST.COM
I20260926 06:27:38.792893 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:38.781645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:49181 (local address 127.2.39.195:44687)
0926 06:27:38.781798 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:38.781806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:38.781833 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:27:38.782371 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:38.782374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:38.782423 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:38.782568 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:38.783892 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:38.785222 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:38.786175 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:38.786781 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:38.786889 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:38.788721 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:38.788725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:38.788725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:38.788740 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:38.790758 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:38.791403 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:38.791406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:38.791406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:38.791520 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:38.791878 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:38.791879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:38.791880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:38.792201 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:38.792294 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:38.792642 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 06:27:38.792777 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260926 06:27:38.792897 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:38.781604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:60739 (local address 127.2.39.194:34313)
0926 06:27:38.781757 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:38.781769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:38.781793 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:27:38.781906 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:38.781907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:38.781948 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:38.782052 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:38.782867 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:38.783741 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:38.785187 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:38.785817 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:38.785928 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:38.788330 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:38.788333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:38.788334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:38.788343 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:38.790754 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:38.791403 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:38.791406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:38.791406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:38.791529 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:38.791883 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:38.791884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:38.791884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:38.792201 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:38.792300 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:38.792642 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 06:27:38.792777 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260926 06:27:38.792886 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:38.781483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:34313 (local address 127.2.39.193:60739)
0926 06:27:38.781617 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:38.781628 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:38.781684 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:38.782114 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:38.782117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:38.782516 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:38.782765 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:38.782770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:38.783842 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:38.783844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:38.785097 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:38.785100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:38.785901 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:38.785902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:38.786018 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:38.786614 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:38.786624 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:38.788233 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:38.790843 (+ 2610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:38.790851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:38.790853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:38.791318 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:38.791594 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:38.791594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:38.791595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:38.791761 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:38.792283 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:38.792290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:38.792442 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 06:27:38.792559 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 06:27:38.792747 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":24,"thread_start_us":45,"threads_started":1}
I20260926 06:27:38.792906 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:38.781463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:44687 (local address 127.2.39.193:49181)
0926 06:27:38.781529 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:38.781542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:38.781623 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:38.782559 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:38.782560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:38.782710 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:38.782767 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:38.782773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:38.785301 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:38.785302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:38.786062 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:38.786065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:38.786861 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:38.786862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:38.786904 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:38.787325 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:38.787329 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:38.788625 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:38.790843 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:38.790851 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:38.790853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:38.791318 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:38.791594 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:38.791594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:38.791595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:38.791761 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:38.792283 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:38.792290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:38.792442 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 06:27:38.792559 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 06:27:38.792799 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":100}
I20260926 06:27:38.793393 17278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802704a03047494e9bce6f8a74145726" candidate_uuid: "3423711e5d5c456abfe9e34a5713238d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" is_pre_election: true
I20260926 06:27:38.793583 17278 raft_consensus.cc:2454] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3423711e5d5c456abfe9e34a5713238d in term 1.
I20260926 06:27:38.793870 17074 leader_election.cc:302] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3423711e5d5c456abfe9e34a5713238d, abe7fcf398a9465c8bbfbe9e9cc323ac; no voters:
W20260926 06:27:38.794005 17076 leader_election.cc:341] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687): Not found: Tablet not found: 802704a03047494e9bce6f8a74145726
I20260926 06:27:38.794011 17464 raft_consensus.cc:2781] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 06:27:38.794121 17464 raft_consensus.cc:482] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:27:38.794153 17464 raft_consensus.cc:3037] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:27:38.795325 17464 raft_consensus.cc:504] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:38.795500 17464 leader_election.cc:288] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [CANDIDATE]: Term 2 election: Requested vote from peers 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687), abe7fcf398a9465c8bbfbe9e9cc323ac (127.2.39.194:34313)
I20260926 06:27:38.795639 17278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "802704a03047494e9bce6f8a74145726" candidate_uuid: "3423711e5d5c456abfe9e34a5713238d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac"
I20260926 06:27:38.795717 17278 raft_consensus.cc:3037] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 1 FOLLOWER]: Advancing to term 2
W20260926 06:27:38.795740 17076 leader_election.cc:341] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687): Not found: Tablet not found: 802704a03047494e9bce6f8a74145726
I20260926 06:27:38.796859 17278 raft_consensus.cc:2454] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3423711e5d5c456abfe9e34a5713238d in term 2.
I20260926 06:27:38.797020 17074 leader_election.cc:302] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3423711e5d5c456abfe9e34a5713238d, abe7fcf398a9465c8bbfbe9e9cc323ac; no voters: 128bd3a164cc460bad90618cdd643b70
I20260926 06:27:38.797135 17464 raft_consensus.cc:2781] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:27:38.797259 17464 raft_consensus.cc:686] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [term 2 LEADER]: Becoming Leader. State: Replica: 3423711e5d5c456abfe9e34a5713238d, State: Running, Role: LEADER
I20260926 06:27:38.797364 17464 consensus_queue.cc:237] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:38.797616 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:38.797442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:60781)
0926 06:27:38.797541 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:38.797588 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260926 06:27:39.356865 17076 consensus_peers.cc:487] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d -> Peer 128bd3a164cc460bad90618cdd643b70 (127.2.39.195:44687): Couldn't send request to peer 128bd3a164cc460bad90618cdd643b70. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 802704a03047494e9bce6f8a74145726. This is attempt 1: this message will repeat every 5th retry.
I20260926 06:27:39.404784 17278 raft_consensus.cc:1261] T 802704a03047494e9bce6f8a74145726 P abe7fcf398a9465c8bbfbe9e9cc323ac [term 2 FOLLOWER]: Refusing update from remote peer 3423711e5d5c456abfe9e34a5713238d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 06:27:39.405196 17470 consensus_queue.cc:1035] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d [LEADER]: Connected to new peer: Peer: permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 06:27:39.406060 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:39.405906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:35291)
0926 06:27:39.405984 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:39.406024 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":41,"threads_started":1}
I20260926 06:27:39.406961 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:39.406785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:42203)
0926 06:27:39.406888 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:39.406922 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":52,"threads_started":1}
I20260926 06:27:39.407007 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:39.406906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:58797)
0926 06:27:39.406942 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:39.406980 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 06:27:39.615761 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:39.615551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:38567)
0926 06:27:39.615688 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:39.615726 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":55,"threads_started":1}
I20260926 06:27:39.830895 17479 ts_tablet_manager.cc:927] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Initiating tablet copy from peer 3423711e5d5c456abfe9e34a5713238d (127.2.39.193:45395)
I20260926 06:27:39.831111 17479 tablet_copy_client.cc:316] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: tablet copy: Beginning tablet copy session from remote peer at address 127.2.39.193:45395
Sep 26 06:27:39 dist-test-slave-fkbn krb5kdc[15887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404057, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.193@KRBTEST.COM
I20260926 06:27:39.839094 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:39.831373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45395 (local address 127.2.39.195:58585)
0926 06:27:39.831434 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:39.831456 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:39.831549 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:39.831762 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:39.831767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:39.832031 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:39.832097 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:39.832099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:39.832911 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:39.832911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:39.833475 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:39.833476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:39.833967 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:39.833968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:39.834026 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:39.834657 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:39.834662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:39.836112 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:39.838023 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:39.838029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:39.838031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:39.838218 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:39.838473 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:39.838474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:39.838474 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:39.838532 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:39.838803 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:39.838809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:39.838856 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:27:39.838930 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:27:39.839016 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 06:27:39.839104 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:39.831475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:58585 (local address 127.2.39.193:45395)
0926 06:27:39.831634 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:39.831638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:39.831648 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:27:39.831671 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:39.831673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:39.831685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:39.831748 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:39.832164 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:39.832847 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:39.833552 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:39.833903 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:39.833960 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:39.836196 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:39.836203 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:39.836204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:39.836210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:39.837952 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:39.838298 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:39.838301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:39.838301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:39.838413 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:39.838592 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:39.838593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:39.838593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:39.838731 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:39.838798 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:39.838970 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 06:27:39.839037 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260926 06:27:39.839594 17162 tablet_copy_service.cc:140] P 3423711e5d5c456abfe9e34a5713238d: Received BeginTabletCopySession request for tablet 802704a03047494e9bce6f8a74145726 from peer 128bd3a164cc460bad90618cdd643b70 ({username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:58585)
I20260926 06:27:39.839673 17162 tablet_copy_service.cc:161] P 3423711e5d5c456abfe9e34a5713238d: Beginning new tablet copy session on tablet 802704a03047494e9bce6f8a74145726 from peer 128bd3a164cc460bad90618cdd643b70 at {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:58585: session id = 128bd3a164cc460bad90618cdd643b70-802704a03047494e9bce6f8a74145726
I20260926 06:27:39.840149 17162 tablet_copy_source_session.cc:205] T 802704a03047494e9bce6f8a74145726 P 3423711e5d5c456abfe9e34a5713238d: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 06:27:39.841061 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 802704a03047494e9bce6f8a74145726. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:39.843562 17479 tablet_copy_client.cc:791] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: tablet copy: Starting download of 0 data blocks...
I20260926 06:27:39.843729 17479 tablet_copy_client.cc:655] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: tablet copy: Starting download of 1 WAL segments...
I20260926 06:27:39.844444 17479 tablet_copy_client.cc:531] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 06:27:39.845111 17479 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Bootstrap starting.
I20260926 06:27:39.846511 17479 log.cc:824] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:39.848461 17479 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:27:39.848690 17479 tablet_bootstrap.cc:492] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Bootstrap complete.
I20260926 06:27:39.848834 17479 ts_tablet_manager.cc:1397] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 06:27:39.849820 17479 raft_consensus.cc:348] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:39.850010 17479 raft_consensus.cc:729] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128bd3a164cc460bad90618cdd643b70, State: Initialized, Role: FOLLOWER
I20260926 06:27:39.850142 17479 consensus_queue.cc:260] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "128bd3a164cc460bad90618cdd643b70" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 44687 } } peers { permanent_uuid: "abe7fcf398a9465c8bbfbe9e9cc323ac" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 34313 } } peers { permanent_uuid: "3423711e5d5c456abfe9e34a5713238d" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45395 } }
I20260926 06:27:39.850394 17479 ts_tablet_manager.cc:1428] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 06:27:39.850620 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:39.850527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:51161)
0926 06:27:39.850563 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:39.850590 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 06:27:39.850663 17162 tablet_copy_service.cc:342] P 3423711e5d5c456abfe9e34a5713238d: Request end of tablet copy session 128bd3a164cc460bad90618cdd643b70-802704a03047494e9bce6f8a74145726 received from {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:58585
I20260926 06:27:39.850723 17162 tablet_copy_service.cc:433] P 3423711e5d5c456abfe9e34a5713238d: ending tablet copy session 128bd3a164cc460bad90618cdd643b70-802704a03047494e9bce6f8a74145726 on tablet 802704a03047494e9bce6f8a74145726 with peer 128bd3a164cc460bad90618cdd643b70
I20260926 06:27:39.850881 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:39.850808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:49077)
0926 06:27:39.850837 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:39.850861 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260926 06:27:40.396940 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:40.396780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:36395)
0926 06:27:40.396867 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:40.396898 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":43,"threads_started":1}
I20260926 06:27:40.396906 17411 raft_consensus.cc:1203] T 802704a03047494e9bce6f8a74145726 P 128bd3a164cc460bad90618cdd643b70 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 06:27:40.397500 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:40.397309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:37749)
0926 06:27:40.397402 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:40.397433 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":39,"threads_started":1}
I20260926 06:27:40.407572 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:40.407333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:40015)
0926 06:27:40.407501 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:40.407536 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":40,"threads_started":1}
I20260926 06:27:41.397553 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:41.397346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:56073)
0926 06:27:41.397473 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:41.397508 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260926 06:27:41.398061 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:41.397874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:60403)
0926 06:27:41.397977 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:41.398007 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260926 06:27:41.408191 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:41.408010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:34955)
0926 06:27:41.408124 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:41.408152 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260926 06:27:42.398272 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:42.398007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:39991)
0926 06:27:42.398190 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:42.398225 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260926 06:27:42.398614 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:42.398447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:45017)
0926 06:27:42.398544 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:42.398578 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":56,"threads_started":1}
I20260926 06:27:42.398703 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:42.398597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:45729)
0926 06:27:42.398642 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:42.398671 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 06:27:42.408732 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:42.408554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:49039)
0926 06:27:42.408663 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:42.408695 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":51,"threads_started":1}
I20260926 06:27:43.399333 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:43.399131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:37161)
0926 06:27:43.399254 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:43.399287 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260926 06:27:43.399358 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:43.399131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:40739)
0926 06:27:43.399286 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:43.399316 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260926 06:27:43.409384 17499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:43.409182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:43657)
0926 06:27:43.409309 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:43.409343 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260926 06:27:44.399981 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:44.399748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:49175)
0926 06:27:44.399885 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:44.399919 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260926 06:27:44.400141 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:44.399930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:51853)
0926 06:27:44.400032 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:44.400087 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
I20260926 06:27:44.410091 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:44.409883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:53933)
0926 06:27:44.409996 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:44.410046 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260926 06:27:45.400771 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:45.400536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:44689)
0926 06:27:45.400687 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:45.400725 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260926 06:27:45.400787 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:45.400574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:57623)
0926 06:27:45.400716 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:45.400744 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":59,"threads_started":1}
I20260926 06:27:45.410670 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:45.410490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:47487)
0926 06:27:45.410595 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:45.410625 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":57,"threads_started":1}
I20260926 06:27:46.401433 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:46.401220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:39903)
0926 06:27:46.401363 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:46.401389 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260926 06:27:46.401433 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:46.401210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:47115)
0926 06:27:46.401364 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:46.401393 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260926 06:27:46.411306 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:46.411103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:54261)
0926 06:27:46.411225 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:46.411261 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260926 06:27:47.402144 17511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:47.401907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:41097)
0926 06:27:47.402066 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:47.402100 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260926 06:27:47.402166 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:47.401965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:47615)
0926 06:27:47.402095 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:47.402126 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260926 06:27:47.411984 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:47.411702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:55183)
0926 06:27:47.411889 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:47.411923 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":52,"threads_started":1}
I20260926 06:27:48.402832 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:48.402601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:41863)
0926 06:27:48.402744 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:48.402781 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260926 06:27:48.402832 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:48.402626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:40209)
0926 06:27:48.402744 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:48.402780 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":51,"threads_started":1}
I20260926 06:27:48.412631 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:48.412408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:40721)
0926 06:27:48.412559 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:48.412588 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260926 06:27:49.403549 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:49.403309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:60917)
0926 06:27:49.403447 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:49.403492 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260926 06:27:49.403591 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:49.403393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:48225)
0926 06:27:49.403517 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:49.403552 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260926 06:27:49.413331 17519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:49.413124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:51735)
0926 06:27:49.413253 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:49.413289 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260926 06:27:50.404428 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:50.404085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:46391)
0926 06:27:50.404331 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:50.404373 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":75,"threads_started":1}
I20260926 06:27:50.404520 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:50.404157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:37097)
0926 06:27:50.404402 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:50.404437 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":63,"threads_started":1}
I20260926 06:27:50.414105 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:50.413890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:50195)
0926 06:27:50.414023 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:50.414061 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260926 06:27:51.405103 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:51.404873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:36209)
0926 06:27:51.405022 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:51.405057 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260926 06:27:51.405189 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:51.404961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:43183)
0926 06:27:51.405095 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:51.405124 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":52,"threads_started":1}
I20260926 06:27:51.414750 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:51.414542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:34737)
0926 06:27:51.414672 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:51.414706 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":55,"threads_started":1}
I20260926 06:27:52.405938 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:52.405725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.193:53235)
0926 06:27:52.405863 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:52.405898 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260926 06:27:52.405932 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:52.405734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.195:60867)
0926 06:27:52.405855 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:52.405887 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260926 06:27:52.415412 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:52.415215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:33781 (local address 127.2.39.194:35683)
0926 06:27:52.415339 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:52.415370 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:33781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260926 06:27:52.628605 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 17029
I20260926 06:27:52.633325 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 17191
I20260926 06:27:52.637611 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 17328
2026-09-26T06:27:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18143 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:27:52 dist-test-slave-fkbn krb5kdc[17531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:27:52 dist-test-slave-fkbn krb5kdc[17531](info): set up 2 sockets
Sep 26 06:27:52 dist-test-slave-fkbn krb5kdc[17531](info): commencing operation
krb5kdc: starting...
W20260926 06:27:53.673667 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:27:53 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:27:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:27:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:27:53.793313 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:42591
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:42591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:53.873756 17544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:53.873952 17544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:53.873974 17544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:53.875085 17544 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 06:27:53.875123 17544 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 06:27:53.875135 17544 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 06:27:53.875146 17544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:27:53.875159 17544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:53.875172 17544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:53.875185 17544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:27:53.875196 17544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:27:53.876605 17544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:42591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:42591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:53.876791 17544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:53.876960 17544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:53.882534 17549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:53.882616 17552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:53.882507 17550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:53.882484 17544 server_base.cc:1034] running on GCE node
I20260926 06:27:53.882977 17544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:53.883270 17544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:53.884410 17544 hybrid_clock.cc:648] HybridClock initialized: now 1790404073884399 us; error 29 us; skew 500 ppm
Sep 26 06:27:53 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404073, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:53.890657 17544 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:27:53.891150 17544 webserver.cc:460] Webserver started at http://127.2.39.254:34285/ using document root <none> and password file <none>
I20260926 06:27:53.891397 17544 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:53.891464 17544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:53.891620 17544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:53.892701 17544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "4e89906280544dea8e1a75f174ee8102"
format_stamp: "Formatted at 2026-09-26 06:27:53 on dist-test-slave-fkbn"
server_key: "29495ab770a1c00fb6cb10156e38748b"
server_key_iv: "3e2a0d19328238b03ad8d689bb95b25c"
server_key_version: "encryptionkey@0"
I20260926 06:27:53.893067 17544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "4e89906280544dea8e1a75f174ee8102"
format_stamp: "Formatted at 2026-09-26 06:27:53 on dist-test-slave-fkbn"
server_key: "29495ab770a1c00fb6cb10156e38748b"
server_key_iv: "3e2a0d19328238b03ad8d689bb95b25c"
server_key_version: "encryptionkey@0"
I20260926 06:27:53.894528 17544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:27:53.895262 17559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:53.895429 17544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:53.895547 17544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "4e89906280544dea8e1a75f174ee8102"
format_stamp: "Formatted at 2026-09-26 06:27:53 on dist-test-slave-fkbn"
server_key: "29495ab770a1c00fb6cb10156e38748b"
server_key_iv: "3e2a0d19328238b03ad8d689bb95b25c"
server_key_version: "encryptionkey@0"
I20260926 06:27:53.895623 17544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:53.923223 17544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:53.925673 17544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:53.925840 17544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:53.930119 17544 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:42591
I20260926 06:27:53.930156 17611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:42591 every 8 connection(s)
I20260926 06:27:53.930560 17544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:27:53.931437 17612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:53.933789 17612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Bootstrap starting.
I20260926 06:27:53.934351 17612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:53.934597 17612 log.cc:824] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:53.935253 17612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: No bootstrap required, opened a new log
I20260926 06:27:53.936793 17612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:27:53.936934 17612 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:53.936957 17612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e89906280544dea8e1a75f174ee8102, State: Initialized, Role: FOLLOWER
I20260926 06:27:53.937109 17612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:27:53.937194 17612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:27:53.937218 17612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:27:53.937247 17612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:53.937814 17612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:27:53.937901 17612 leader_election.cc:302] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e89906280544dea8e1a75f174ee8102; no voters:
I20260926 06:27:53.938093 17612 leader_election.cc:288] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:27:53.938127 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 17544
I20260926 06:27:53.938148 17615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:27:53.938234 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:27:53.938300 17615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 1 LEADER]: Becoming Leader. State: Replica: 4e89906280544dea8e1a75f174ee8102, State: Running, Role: LEADER
I20260926 06:27:53.938375 17612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:27:53.938441 2207 external_mini_cluster.cc:1428] Setting key 0363709d5a8bea259ce13a3f44125ea1
I20260926 06:27:53.938380 17615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:27:53.938886 17616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e89906280544dea8e1a75f174ee8102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } } }
I20260926 06:27:53.938891 17617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e89906280544dea8e1a75f174ee8102. Latest consensus state: current_term: 1 leader_uuid: "4e89906280544dea8e1a75f174ee8102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } } }
I20260926 06:27:53.938979 17617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:53.938977 17616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: This master's current role is: LEADER
I20260926 06:27:53.939843 17621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:27:53.940428 17621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:27:53.942011 17621 catalog_manager.cc:1347] Generated new cluster ID: 7cfdf676bb56481383e513f18bc4a9ba
I20260926 06:27:53.942063 17621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:27:53 dist-test-slave-fkbn krb5kdc[17531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:53.949647 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:53.939290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47684 (local address 127.2.39.254:42591)
0926 06:27:53.939428 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:53.939439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:53.939614 (+ 175us) server_negotiation.cc:407] Connection header received
0926 06:27:53.939744 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:53.939753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:53.939824 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:53.940729 (+ 905us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:53.940802 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:53.942174 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:53.942959 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:53.943392 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:53.943510 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:53.945873 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:53.945877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:53.945877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:53.945889 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:53.948075 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:53.948446 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:53.948449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:53.948450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:53.948545 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:53.948763 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:53.948764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:53.948764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:53.948996 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:53.949166 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:53.949241 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 06:27:53.949383 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
I20260926 06:27:53.973816 17621 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:27:53.974519 17621 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:27:53.981277 17621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Generated new TSK 0
I20260926 06:27:53.981519 17621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:27:54.016289 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:42591
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:54.121097 17638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:54.121291 17638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:54.121313 17638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:54.122525 17638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:54.122570 17638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:54.122620 17638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:27:54.124058 17638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:42591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:54.124289 17638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:54.124534 17638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:54.129742 17643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:54.129751 17644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:54.129743 17646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:54.129760 17638 server_base.cc:1034] running on GCE node
I20260926 06:27:54.130182 17638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:54.130388 17638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:54.131567 17638 hybrid_clock.cc:648] HybridClock initialized: now 1790404074131556 us; error 28 us; skew 500 ppm
Sep 26 06:27:54 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:54.134217 17638 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:27:54.134717 17638 webserver.cc:460] Webserver started at http://127.2.39.193:35585/ using document root <none> and password file <none>
I20260926 06:27:54.134905 17638 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:54.134944 17638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:54.135044 17638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:54.136060 17638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "90cf626a84c644dfba730fa44d69e3e7"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "f3b814d2450a63ae99ded19d649563a6"
server_key_iv: "81698ac66a0d96a59d49d26e3ada284f"
server_key_version: "encryptionkey@0"
I20260926 06:27:54.136359 17638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "90cf626a84c644dfba730fa44d69e3e7"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "f3b814d2450a63ae99ded19d649563a6"
server_key_iv: "81698ac66a0d96a59d49d26e3ada284f"
server_key_version: "encryptionkey@0"
I20260926 06:27:54.137755 17638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 06:27:54.138458 17653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:54.138680 17638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:54.138792 17638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "90cf626a84c644dfba730fa44d69e3e7"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "f3b814d2450a63ae99ded19d649563a6"
server_key_iv: "81698ac66a0d96a59d49d26e3ada284f"
server_key_version: "encryptionkey@0"
I20260926 06:27:54.138875 17638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:54.151337 17638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:54.153767 17638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:54.153934 17638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:54.154177 17638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:54.154495 17638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:54.154551 17638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:54.154592 17638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:54.154619 17638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:54.160727 17638 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:33235
I20260926 06:27:54.160786 17766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:33235 every 8 connection(s)
I20260926 06:27:54.161128 17638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:27:54.161782 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 17638
I20260926 06:27:54.161898 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:27:54.162164 2207 external_mini_cluster.cc:1428] Setting key d9923ef86f204984b3f4fbb74ebf498c
Sep 26 06:27:54 dist-test-slave-fkbn krb5kdc[17531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
I20260926 06:27:54.171455 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.162180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:47423 (local address 127.2.39.254:42591)
0926 06:27:54.162248 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:54.162250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:54.162265 (+ 15us) server_negotiation.cc:407] Connection header received
0926 06:27:54.162818 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:54.162819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:54.162831 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:54.162884 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:54.163848 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.164397 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.165820 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.166031 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.166091 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.168540 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:54.168541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:54.168542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:54.168549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:54.170036 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:54.170460 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:54.170462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:54.170462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:54.170529 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:54.170783 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:54.170783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:54.170784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:54.170901 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:54.170979 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:54.171264 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 06:27:54.171362 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:27:54.171630 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.161938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:47423)
0926 06:27:54.162133 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:54.162183 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:54.162707 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:54.162973 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:54.162976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:54.163549 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:54.163747 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.163751 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.164500 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.164502 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.165739 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.165742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.166122 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.166124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.166280 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.166838 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:54.166850 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:54.168436 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:54.170136 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:54.170143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:54.170145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:54.170368 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:54.170598 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:54.170599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:54.170599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:54.170713 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:54.170970 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:54.170977 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:54.171124 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 06:27:54.171230 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 06:27:54.171389 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260926 06:27:54.172243 17767 heartbeater.cc:347] Connected to a master server at 127.2.39.254:42591
I20260926 06:27:54.172358 17767 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:54.172547 17767 heartbeater.cc:511] Master 127.2.39.254:42591 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
I20260926 06:27:54.173125 17576 ts_manager.cc:194] Registered new tserver with Master: 90cf626a84c644dfba730fa44d69e3e7 (127.2.39.193:33235)
I20260926 06:27:54.173832 17576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:47423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:27:54.188371 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:42591
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:54.270058 17772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:54.270248 17772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:54.270272 17772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:54.271420 17772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:54.271467 17772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:54.271523 17772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:27:54.273000 17772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:42591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:54.273239 17772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:54.273491 17772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:54.278391 17780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:54.278391 17778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:54.278522 17777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:54.278735 17772 server_base.cc:1034] running on GCE node
I20260926 06:27:54.278942 17772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:54.279182 17772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:54.280323 17772 hybrid_clock.cc:648] HybridClock initialized: now 1790404074280307 us; error 30 us; skew 500 ppm
Sep 26 06:27:54 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:54.282871 17772 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:27:54.283340 17772 webserver.cc:460] Webserver started at http://127.2.39.194:42061/ using document root <none> and password file <none>
I20260926 06:27:54.283591 17772 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:54.283661 17772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:54.283809 17772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:54.284898 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "9c969493567644d39e8a4bc024eb1bfc"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "e6eff7bbb68ed83b43607a0a4d80e6bc"
server_key_iv: "e056660f9d456f3ba8a6ba77e3717b45"
server_key_version: "encryptionkey@0"
I20260926 06:27:54.285248 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "9c969493567644d39e8a4bc024eb1bfc"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "e6eff7bbb68ed83b43607a0a4d80e6bc"
server_key_iv: "e056660f9d456f3ba8a6ba77e3717b45"
server_key_version: "encryptionkey@0"
I20260926 06:27:54.286610 17772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:27:54.287302 17787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:54.287472 17772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:27:54.287568 17772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "9c969493567644d39e8a4bc024eb1bfc"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "e6eff7bbb68ed83b43607a0a4d80e6bc"
server_key_iv: "e056660f9d456f3ba8a6ba77e3717b45"
server_key_version: "encryptionkey@0"
I20260926 06:27:54.287650 17772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:54.298228 17772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:54.300055 17772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:54.300215 17772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:54.300429 17772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:54.300733 17772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:54.300808 17772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:54.300843 17772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:54.300870 17772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:54.306650 17772 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:36829
I20260926 06:27:54.306689 17900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:36829 every 8 connection(s)
I20260926 06:27:54.307015 17772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:27:54 dist-test-slave-fkbn krb5kdc[17531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:54.313313 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 17772
I20260926 06:27:54.313444 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:27:54.313704 2207 external_mini_cluster.cc:1428] Setting key ccc5dd919ca4f211694a502067aacc96
I20260926 06:27:54.316259 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.307702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:58017 (local address 127.2.39.254:42591)
0926 06:27:54.307787 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:54.307789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:54.307950 (+ 161us) server_negotiation.cc:407] Connection header received
0926 06:27:54.308443 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:54.308445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:54.308454 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:54.308522 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:54.309419 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.309961 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.310919 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.311101 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.311157 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.313322 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:54.313323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:54.313323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:54.313329 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:54.314857 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:54.315293 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:54.315294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:54.315294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:54.315350 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:54.315598 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:54.315599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:54.315599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:54.315707 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:54.315777 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:54.316064 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 06:27:54.316148 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 06:27:54.316474 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.307661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:58017)
0926 06:27:54.307838 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:54.307874 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:54.308351 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:54.308581 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:54.308584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:54.309144 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:54.309335 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.309340 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.310051 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.310052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.310844 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.310845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.311175 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.311176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.311281 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.311816 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:54.311825 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:54.313227 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:54.314946 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:54.314953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:54.314956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:54.315206 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:54.315423 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:54.315424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:54.315424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:54.315527 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:54.315779 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:54.315786 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:54.315921 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 06:27:54.316054 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 06:27:54.316237 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":94,"threads_started":1}
I20260926 06:27:54.316970 17901 heartbeater.cc:347] Connected to a master server at 127.2.39.254:42591
I20260926 06:27:54.317075 17901 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:54.317267 17901 heartbeater.cc:511] Master 127.2.39.254:42591 requested a full tablet report, sending...
I20260926 06:27:54.317672 17576 ts_manager.cc:194] Registered new tserver with Master: 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829)
I20260926 06:27:54.318216 17576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:58017
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:27:54.340075 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:42591
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:27:54.420502 17906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:27:54.420702 17906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:27:54.420724 17906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:27:54.421795 17906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:27:54.421828 17906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:27:54.421854 17906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:27:54.423173 17906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:42591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:27:54.423343 17906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:27:54.423506 17906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:27:54.428524 17912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:54.428581 17911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:27:54.428507 17914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:27:54.428755 17906 server_base.cc:1034] running on GCE node
I20260926 06:27:54.428972 17906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:27:54.429205 17906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:27:54.430356 17906 hybrid_clock.cc:648] HybridClock initialized: now 1790404074430353 us; error 35 us; skew 500 ppm
Sep 26 06:27:54 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:27:54.432761 17906 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:27:54.433163 17906 webserver.cc:460] Webserver started at http://127.2.39.195:35387/ using document root <none> and password file <none>
I20260926 06:27:54.433379 17906 fs_manager.cc:359] Metadata directory not provided
I20260926 06:27:54.433445 17906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:27:54.433636 17906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:27:54.434648 17906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "41f3cab812394cb68e97440801aaea62"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "160ef42dd51a91eb0e27e319c3d2317a"
server_key_iv: "63d2382e96a1fd5bad5c43532572106c"
server_key_version: "encryptionkey@0"
I20260926 06:27:54.435075 17906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "41f3cab812394cb68e97440801aaea62"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "160ef42dd51a91eb0e27e319c3d2317a"
server_key_iv: "63d2382e96a1fd5bad5c43532572106c"
server_key_version: "encryptionkey@0"
I20260926 06:27:54.436342 17906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:27:54.437039 17921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:54.437207 17906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:27:54.437302 17906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "41f3cab812394cb68e97440801aaea62"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "160ef42dd51a91eb0e27e319c3d2317a"
server_key_iv: "63d2382e96a1fd5bad5c43532572106c"
server_key_version: "encryptionkey@0"
I20260926 06:27:54.437382 17906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:27:54.452215 17906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:27:54.454185 17906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:27:54.454344 17906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:27:54.454584 17906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:27:54.454895 17906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:27:54.454947 17906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:54.454980 17906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:27:54.455001 17906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:27:54.460681 17906 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:43493
I20260926 06:27:54.460728 18034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:43493 every 8 connection(s)
I20260926 06:27:54.461030 17906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:27:54.465270 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 17906
I20260926 06:27:54.465370 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:27:54.465602 2207 external_mini_cluster.cc:1428] Setting key 3c24de07ff30bbc1240dc933e9f81b50
Sep 26 06:27:54 dist-test-slave-fkbn krb5kdc[17531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404074, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:27:54.470742 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.461753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:40511 (local address 127.2.39.254:42591)
0926 06:27:54.461825 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:54.461828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:54.462028 (+ 200us) server_negotiation.cc:407] Connection header received
0926 06:27:54.462499 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:54.462500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:54.462510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:54.462559 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:54.463547 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.464041 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.465009 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.465426 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.465514 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.468065 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:54.468066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:54.468067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:54.468075 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:54.469261 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:54.469751 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:54.469753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:54.469753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:54.469819 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:54.470079 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:54.470080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:54.470080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:54.470204 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:54.470276 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:54.470570 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 06:27:54.470650 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:27:54.470943 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.461743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:40511)
0926 06:27:54.461910 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:54.461945 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:54.462402 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:54.462623 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:54.462627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:54.463252 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:27:54.463447 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.463451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.464123 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.464124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.464912 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.464914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.465518 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.465519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.465698 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.466368 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:27:54.466380 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:27:54.467961 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:27:54.469348 (+ 1387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:54.469354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:54.469357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:54.469674 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:54.469896 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:27:54.469897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:27:54.469897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:27:54.470011 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:27:54.470281 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:27:54.470287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:27:54.470425 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 06:27:54.470535 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 06:27:54.470699 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
I20260926 06:27:54.471415 18035 heartbeater.cc:347] Connected to a master server at 127.2.39.254:42591
I20260926 06:27:54.471516 18035 heartbeater.cc:464] Registering TS with master...
I20260926 06:27:54.471700 18035 heartbeater.cc:511] Master 127.2.39.254:42591 requested a full tablet report, sending...
I20260926 06:27:54.472090 17576 ts_manager.cc:194] Registered new tserver with Master: 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493)
I20260926 06:27:54.472474 17576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:40511
I20260926 06:27:54.477862 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:27:54.485214 17626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.479292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47698 (local address 127.2.39.254:42591)
0926 06:27:54.479330 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:54.479332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:54.479448 (+ 116us) server_negotiation.cc:407] Connection header received
0926 06:27:54.479499 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:54.479500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:54.479518 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:54.479556 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:27:54.480045 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.480524 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.481265 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.481483 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.481543 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.482853 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:27:54.482855 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:27:54.482855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:27:54.482861 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:27:54.484246 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:54.484584 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:54.484585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:54.484586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:54.484640 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:27:54.484802 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:27:54.484803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:27:54.484803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:27:54.484931 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:27:54.485001 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:54.485092 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 06:27:54.485152 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:27:54.486790 17576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:27:54.488385 17576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:27:54.494999 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.490928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:36829 (local address 127.0.0.1:58040)
0926 06:27:54.491139 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:54.491149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:54.491189 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:54.491428 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:54.491431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:54.491434 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:54.491525 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.491528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.492848 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.492849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.494000 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.494002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.494698 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.494699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.494774 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.494787 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 06:27:54.494866 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:27:54.494921 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260926 06:27:54.495013 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.491052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.2.39.194:36829)
0926 06:27:54.491250 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:54.491255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:54.491264 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:27:54.491323 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:54.491325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:54.491339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:54.491410 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:54.491589 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.492739 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.494105 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.494605 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.494698 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.494761 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:54.494882 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 06:27:54.494922 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":81,"threads_started":1}
I20260926 06:27:54.495229 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.490916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.2.39.193:33235)
0926 06:27:54.491096 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:54.491102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:54.491259 (+ 157us) server_negotiation.cc:407] Connection header received
0926 06:27:54.491320 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:54.491323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:54.491339 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:54.491417 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:54.491718 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.492779 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.493961 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.494699 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.494774 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.494841 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:54.495113 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 06:27:54.495157 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":72,"threads_started":1}
I20260926 06:27:54.495499 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.490810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59228 (local address 127.2.39.195:43493)
0926 06:27:54.491551 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:54.491555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:54.491565 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:27:54.491602 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:54.491604 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:54.491617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:54.491711 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:54.491815 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.492537 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.494088 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.494743 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.494801 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.494853 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:54.495368 (+ 515us) server_negotiation.cc:299] Negotiation successful
0926 06:27:54.495415 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":96,"threads_started":1}
I20260926 06:27:54.495664 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.490722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:33235 (local address 127.0.0.1:45226)
0926 06:27:54.491048 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:54.491092 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:54.491165 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:54.491604 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:54.491605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:54.491607 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:54.491645 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.491646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.492882 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.492883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.493876 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.493878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.494879 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.494880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.494958 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.494961 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:27:54.495016 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:27:54.495052 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260926 06:27:54.496004 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.490722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:43493 (local address 127.0.0.1:59228)
0926 06:27:54.491050 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:54.491091 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:54.491166 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:54.491711 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:54.491712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:54.491714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:54.491758 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.491759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.492640 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.492641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.494000 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.494002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.494879 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.494880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.494950 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.494955 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:27:54.495006 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:27:54.495052 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":10,"thread_start_us":66,"threads_started":1}
I20260926 06:27:54.495994 17835 tablet_service.cc:1467] Processing CreateTablet for tablet f81457c0de574b52be08f262becd9b40 (DEFAULT_TABLE table=test-table [id=3f9233b6cd66417dbcc77c5414bf7417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:54.496366 17835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f81457c0de574b52be08f262becd9b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:54.496358 17701 tablet_service.cc:1467] Processing CreateTablet for tablet f81457c0de574b52be08f262becd9b40 (DEFAULT_TABLE table=test-table [id=3f9233b6cd66417dbcc77c5414bf7417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:54.496685 17701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f81457c0de574b52be08f262becd9b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:54.498317 17969 tablet_service.cc:1467] Processing CreateTablet for tablet f81457c0de574b52be08f262becd9b40 (DEFAULT_TABLE table=test-table [id=3f9233b6cd66417dbcc77c5414bf7417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:27:54.498658 17969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f81457c0de574b52be08f262becd9b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:27:54.499656 18052 tablet_bootstrap.cc:492] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: Bootstrap starting.
I20260926 06:27:54.499861 18053 tablet_bootstrap.cc:492] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Bootstrap starting.
I20260926 06:27:54.500355 18052 tablet_bootstrap.cc:654] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:54.500437 18053 tablet_bootstrap.cc:654] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:54.501030 18052 log.cc:824] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:54.501053 18053 log.cc:824] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:54.501564 18056 tablet_bootstrap.cc:492] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: Bootstrap starting.
I20260926 06:27:54.501995 18053 tablet_bootstrap.cc:492] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: No bootstrap required, opened a new log
I20260926 06:27:54.502071 18052 tablet_bootstrap.cc:492] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: No bootstrap required, opened a new log
I20260926 06:27:54.502084 18053 ts_tablet_manager.cc:1397] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:27:54.502141 18052 ts_tablet_manager.cc:1397] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:27:54.502285 18056 tablet_bootstrap.cc:654] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: Neither blocks nor log segments found. Creating new log.
I20260926 06:27:54.502920 18056 log.cc:824] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: Log is configured to *not* fsync() on all Append() calls
I20260926 06:27:54.503695 18056 tablet_bootstrap.cc:492] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: No bootstrap required, opened a new log
I20260926 06:27:54.503764 18056 ts_tablet_manager.cc:1397] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:27:54.503893 18053 raft_consensus.cc:348] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:27:54.504065 18053 raft_consensus.cc:374] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:54.504108 18053 raft_consensus.cc:729] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90cf626a84c644dfba730fa44d69e3e7, State: Initialized, Role: FOLLOWER
I20260926 06:27:54.504237 18053 consensus_queue.cc:260] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:27:54.504237 18052 raft_consensus.cc:348] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:27:54.504390 18052 raft_consensus.cc:374] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:54.504418 18052 raft_consensus.cc:729] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c969493567644d39e8a4bc024eb1bfc, State: Initialized, Role: FOLLOWER
I20260926 06:27:54.504509 18053 ts_tablet_manager.cc:1428] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:27:54.504518 17767 heartbeater.cc:503] Master 127.2.39.254:42591 was elected leader, sending a full tablet report...
I20260926 06:27:54.504524 18052 consensus_queue.cc:260] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:27:54.504781 18052 ts_tablet_manager.cc:1428] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:27:54.504788 17901 heartbeater.cc:503] Master 127.2.39.254:42591 was elected leader, sending a full tablet report...
I20260926 06:27:54.506224 18056 raft_consensus.cc:348] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:27:54.506350 18056 raft_consensus.cc:374] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:27:54.506371 18056 raft_consensus.cc:729] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f3cab812394cb68e97440801aaea62, State: Initialized, Role: FOLLOWER
I20260926 06:27:54.506497 18056 consensus_queue.cc:260] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:27:54.506753 18056 ts_tablet_manager.cc:1428] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 06:27:54.506779 18035 heartbeater.cc:503] Master 127.2.39.254:42591 was elected leader, sending a full tablet report...
W20260926 06:27:54.557615 17902 tablet.cc:2369] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:27:54.661909 17768 tablet.cc:2369] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:27:54.691962 18058 raft_consensus.cc:482] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:27:54.692094 18058 raft_consensus.cc:504] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:27:54.692448 18058 leader_election.cc:288] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493), 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829)
I20260926 06:27:54.695582 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.692766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:39453 (local address 127.2.39.195:43493)
0926 06:27:54.692834 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:54.692841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:54.692894 (+ 53us) server_negotiation.cc:407] Connection header received
0926 06:27:54.692952 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:54.692953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:54.692966 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:54.693101 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:54.693287 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.693840 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.694843 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.695223 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.695310 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.695318 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:54.695466 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 06:27:54.695504 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:27:54.695583 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.692968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:57925 (local address 127.2.39.194:36829)
0926 06:27:54.693021 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:54.693024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:54.693051 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:27:54.693113 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:54.693114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:54.693126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:54.693176 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:27:54.693341 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.693863 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.694837 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.695219 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.695303 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.695311 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:54.695471 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 06:27:54.695514 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 06:27:54.695595 18062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.692878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:36829 (local address 127.2.39.193:57925)
0926 06:27:54.693002 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:54.693012 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:54.693057 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:54.693178 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:54.693178 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:54.693180 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:54.693265 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.693267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.693941 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.693942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.694744 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.694746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.695302 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.695303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.695381 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.695384 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:27:54.695447 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:27:54.695493 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260926 06:27:54.695617 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.692677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:43493 (local address 127.2.39.193:39453)
0926 06:27:54.692812 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:54.692824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:27:54.692891 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:27:54.693111 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:27:54.693112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:27:54.693114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:27:54.693192 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.693194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.693924 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.693925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.694742 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:27:54.694744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.695308 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.695309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:27:54.695386 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.695390 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:27:54.695459 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:27:54.695508 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":21,"thread_start_us":87,"threads_started":1}
I20260926 06:27:54.695981 17855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81457c0de574b52be08f262becd9b40" candidate_uuid: "90cf626a84c644dfba730fa44d69e3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c969493567644d39e8a4bc024eb1bfc" is_pre_election: true
I20260926 06:27:54.696029 17989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81457c0de574b52be08f262becd9b40" candidate_uuid: "90cf626a84c644dfba730fa44d69e3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f3cab812394cb68e97440801aaea62" is_pre_election: true
I20260926 06:27:54.696135 17855 raft_consensus.cc:2454] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90cf626a84c644dfba730fa44d69e3e7 in term 0.
I20260926 06:27:54.696170 17989 raft_consensus.cc:2454] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90cf626a84c644dfba730fa44d69e3e7 in term 0.
I20260926 06:27:54.696367 17657 leader_election.cc:302] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90cf626a84c644dfba730fa44d69e3e7, 9c969493567644d39e8a4bc024eb1bfc; no voters:
I20260926 06:27:54.696529 18058 raft_consensus.cc:2781] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:27:54.696604 18058 raft_consensus.cc:482] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:27:54.696620 18058 raft_consensus.cc:3037] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:54.697350 18058 raft_consensus.cc:504] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:27:54.697481 18058 leader_election.cc:288] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 1 election: Requested vote from peers 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493), 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829)
I20260926 06:27:54.697685 17989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81457c0de574b52be08f262becd9b40" candidate_uuid: "90cf626a84c644dfba730fa44d69e3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f3cab812394cb68e97440801aaea62"
I20260926 06:27:54.697710 17855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f81457c0de574b52be08f262becd9b40" candidate_uuid: "90cf626a84c644dfba730fa44d69e3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c969493567644d39e8a4bc024eb1bfc"
I20260926 06:27:54.697788 17989 raft_consensus.cc:3037] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:54.697816 17855 raft_consensus.cc:3037] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:27:54.698477 17855 raft_consensus.cc:2454] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90cf626a84c644dfba730fa44d69e3e7 in term 1.
I20260926 06:27:54.698644 17657 leader_election.cc:302] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90cf626a84c644dfba730fa44d69e3e7, 9c969493567644d39e8a4bc024eb1bfc; no voters:
I20260926 06:27:54.698686 17989 raft_consensus.cc:2454] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90cf626a84c644dfba730fa44d69e3e7 in term 1.
I20260926 06:27:54.698757 18058 raft_consensus.cc:2781] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:27:54.698917 18058 raft_consensus.cc:686] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 LEADER]: Becoming Leader. State: Replica: 90cf626a84c644dfba730fa44d69e3e7, State: Running, Role: LEADER
I20260926 06:27:54.699014 18058 consensus_queue.cc:237] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:27:54.699779 17576 catalog_manager.cc:5495] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90cf626a84c644dfba730fa44d69e3e7 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "90cf626a84c644dfba730fa44d69e3e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } health_report { overall_health: UNKNOWN } } }
W20260926 06:27:54.711511 18036 tablet.cc:2369] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:27:54.724826 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.722623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.2.39.193:33235)
0926 06:27:54.722687 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 06:27:54.722688 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:27:54.722750 (+ 62us) server_negotiation.cc:407] Connection header received
0926 06:27:54.722804 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:27:54.722805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:27:54.722823 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:27:54.722893 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:27:54.723016 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.723473 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.724221 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:27:54.724398 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:27:54.724464 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:27:54.724572 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:27:54.724667 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:27:54.724725 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:27:54.724757 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 06:27:54.724773 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 06:27:54.726720 17989 raft_consensus.cc:1261] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62 [term 1 FOLLOWER]: Refusing update from remote peer 90cf626a84c644dfba730fa44d69e3e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:27:54.726720 17855 raft_consensus.cc:1261] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc [term 1 FOLLOWER]: Refusing update from remote peer 90cf626a84c644dfba730fa44d69e3e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:27:54.727109 18063 consensus_queue.cc:1035] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:27:54.727237 18058 consensus_queue.cc:1035] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:27:54.730026 18068 mvcc.cc:205] Tried to move back new op lower bound from 7333495090077609984 to 7333495089967710208. Current Snapshot: MvccSnapshot[applied={T|T < 7333495090077609984}]
I20260926 06:27:54.730288 18067 mvcc.cc:205] Tried to move back new op lower bound from 7333495090077609984 to 7333495089967710208. Current Snapshot: MvccSnapshot[applied={T|T < 7333495090077609984}]
I20260926 06:27:54.731611 17576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:27:54.731700 17576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:27:54.732390 17576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Sending DeleteTablet for 3 replicas of tablet f81457c0de574b52be08f262becd9b40
I20260926 06:27:54.732718 17701 tablet_service.cc:1514] Processing DeleteTablet for tablet f81457c0de574b52be08f262becd9b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:54 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:45226
I20260926 06:27:54.732756 17835 tablet_service.cc:1514] Processing DeleteTablet for tablet f81457c0de574b52be08f262becd9b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:54 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:58040
I20260926 06:27:54.732951 17969 tablet_service.cc:1514] Processing DeleteTablet for tablet f81457c0de574b52be08f262becd9b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:54 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:59228
I20260926 06:27:54.732970 18076 tablet_replica.cc:330] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: stopping tablet replica
I20260926 06:27:54.733068 18076 raft_consensus.cc:2229] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:27:54.733151 18077 tablet_replica.cc:330] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: stopping tablet replica
I20260926 06:27:54.733184 18076 raft_consensus.cc:2258] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:54.733201 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 17638
I20260926 06:27:54.733264 18077 raft_consensus.cc:2229] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:27:54.733392 18077 raft_consensus.cc:2258] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:54.733855 18077 ts_tablet_manager.cc:1905] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:27:54.733870 18075 tablet_replica.cc:330] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: stopping tablet replica
I20260926 06:27:54.733989 18075 raft_consensus.cc:2229] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:27:54.734122 18075 raft_consensus.cc:2258] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:27:54.734598 18075 ts_tablet_manager.cc:1905] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:27:54.735836 18077 ts_tablet_manager.cc:1918] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:27:54.735924 18077 log.cc:1196] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/f81457c0de574b52be08f262becd9b40
I20260926 06:27:54.736042 18075 ts_tablet_manager.cc:1918] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:27:54.736136 18075 log.cc:1196] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/f81457c0de574b52be08f262becd9b40
I20260926 06:27:54.736215 18077 ts_tablet_manager.cc:1939] T f81457c0de574b52be08f262becd9b40 P 41f3cab812394cb68e97440801aaea62: Deleting consensus metadata
I20260926 06:27:54.736464 18075 ts_tablet_manager.cc:1939] T f81457c0de574b52be08f262becd9b40 P 9c969493567644d39e8a4bc024eb1bfc: Deleting consensus metadata
I20260926 06:27:54.736739 17563 catalog_manager.cc:4841] TS 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493): tablet f81457c0de574b52be08f262becd9b40 (table test-table [id=3f9233b6cd66417dbcc77c5414bf7417]) successfully deleted
I20260926 06:27:54.736892 17563 catalog_manager.cc:4841] TS 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829): tablet f81457c0de574b52be08f262becd9b40 (table test-table [id=3f9233b6cd66417dbcc77c5414bf7417]) successfully deleted
W20260926 06:27:54.740495 17561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:33235 (error 108)
I20260926 06:27:54.740687 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 17772
I20260926 06:27:54.741297 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:27:54.740862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:33235 (local address 127.0.0.1:45254)
0926 06:27:54.741227 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0926 06:27:54.741270 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:33235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345}
W20260926 06:27:54.741422 17561 catalog_manager.cc:4584] TS 90cf626a84c644dfba730fa44d69e3e7 (127.2.39.193:33235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f81457c0de574b52be08f262becd9b40: Network error: Client connection negotiation failed: client connection to 127.2.39.193:33235: connect: Connection refused (error 111)
I20260926 06:27:54.746529 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 17906
I20260926 06:27:54.750861 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 17544
I20260926 06:28:14.756330 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:42591
--webserver_interface=127.2.39.254
--webserver_port=34285
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:42591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:14.837481 18079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:14.837699 18079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:14.837721 18079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:14.838798 18079 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 06:28:14.838833 18079 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 06:28:14.838846 18079 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 06:28:14.838865 18079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:28:14.838879 18079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:14.838892 18079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:14.838909 18079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:28:14.838927 18079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:28:14.840312 18079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:42591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:42591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=34285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:14.840492 18079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:14.840699 18079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:14.849607 18085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:14.849689 18086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:14.849742 18079 server_base.cc:1034] running on GCE node
W20260926 06:28:14.849607 18088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:14.850198 18079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:14.850494 18079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:14.851643 18079 hybrid_clock.cc:648] HybridClock initialized: now 1790404094851605 us; error 55 us; skew 500 ppm
Sep 26 06:28:14 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404094, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:14.861076 18079 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:28:14.861462 18079 webserver.cc:460] Webserver started at http://127.2.39.254:34285/ using document root <none> and password file <none>
I20260926 06:28:14.861729 18079 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:14.861812 18079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:14.863224 18079 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:28:14.863921 18095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:14.864125 18079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:28:14.864229 18079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "4e89906280544dea8e1a75f174ee8102"
format_stamp: "Formatted at 2026-09-26 06:27:53 on dist-test-slave-fkbn"
server_key: "29495ab770a1c00fb6cb10156e38748b"
server_key_iv: "3e2a0d19328238b03ad8d689bb95b25c"
server_key_version: "encryptionkey@0"
I20260926 06:28:14.864619 18079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:14.879914 18079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:14.881863 18079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:14.882032 18079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:14.886037 18079 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:42591
I20260926 06:28:14.886070 18147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:42591 every 8 connection(s)
I20260926 06:28:14.886449 18079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:28:14.887882 18148 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:28:14.888490 18148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Bootstrap starting.
I20260926 06:28:14.891425 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 18079
I20260926 06:28:14.891697 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:33235
--local_ip_for_outbound_sockets=127.2.39.193
--tserver_master_addrs=127.2.39.254:42591
--webserver_port=35585
--webserver_interface=127.2.39.193
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:28:14.898658 18148 log.cc:824] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:14.902057 18148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:28:14.902576 18148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Bootstrap complete.
I20260926 06:28:14.906253 18148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:28:14.906564 18148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e89906280544dea8e1a75f174ee8102, State: Initialized, Role: FOLLOWER
I20260926 06:28:14.906715 18148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:28:14.906777 18148 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:28:14.906817 18148 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:28:14.906863 18148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 1 FOLLOWER]: Advancing to term 2
I20260926 06:28:14.908129 18148 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:28:14.908236 18148 leader_election.cc:302] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e89906280544dea8e1a75f174ee8102; no voters:
I20260926 06:28:14.908393 18148 leader_election.cc:288] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 06:28:14.908486 18152 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 06:28:14.908638 18152 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 2 LEADER]: Becoming Leader. State: Replica: 4e89906280544dea8e1a75f174ee8102, State: Running, Role: LEADER
I20260926 06:28:14.908694 18148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:28:14.908759 18152 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:28:14.909089 18153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4e89906280544dea8e1a75f174ee8102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } } }
I20260926 06:28:14.909168 18153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:14.909418 18154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e89906280544dea8e1a75f174ee8102. Latest consensus state: current_term: 2 leader_uuid: "4e89906280544dea8e1a75f174ee8102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } } }
I20260926 06:28:14.909545 18154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:14.909621 18158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:28:14.910356 18158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:28:14.910518 18158 catalog_manager.cc:1259] Loaded cluster ID: 7cfdf676bb56481383e513f18bc4a9ba
I20260926 06:28:14.910750 18158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:28:14.911911 18158 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:28:14.912185 18158 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Loaded TSK: 0
I20260926 06:28:14.925071 18158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Generated new TSK 1
I20260926 06:28:14.925172 18158 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:28:14.980929 18150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:14.981153 18150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:14.981190 18150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:14.982403 18150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:14.982468 18150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:14.982513 18150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:28:14.984001 18150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:33235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=35585
--tserver_master_addrs=127.2.39.254:42591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:14.984261 18150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:14.984488 18150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:14.989668 18176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:14.989777 18178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:14.989665 18175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:14.990103 18150 server_base.cc:1034] running on GCE node
I20260926 06:28:14.990329 18150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:14.990618 18150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:14.991782 18150 hybrid_clock.cc:648] HybridClock initialized: now 1790404094991764 us; error 35 us; skew 500 ppm
Sep 26 06:28:14 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404094, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:14.994582 18150 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:28:14.995000 18150 webserver.cc:460] Webserver started at http://127.2.39.193:35585/ using document root <none> and password file <none>
I20260926 06:28:14.995254 18150 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:14.995349 18150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:14.996734 18150 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:28:14.997478 18185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:14.997675 18150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:28:14.997773 18150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "90cf626a84c644dfba730fa44d69e3e7"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "f3b814d2450a63ae99ded19d649563a6"
server_key_iv: "81698ac66a0d96a59d49d26e3ada284f"
server_key_version: "encryptionkey@0"
I20260926 06:28:14.998126 18150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:15.032298 18150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:15.034806 18150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:15.035001 18150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:15.035262 18150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:15.035692 18192 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:28:15.037786 18150 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:28:15.037859 18150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:28:15.037909 18150 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:28:15.038560 18150 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:28:15.038617 18150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 06:28:15.038692 18192 tablet_bootstrap.cc:492] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Bootstrap starting.
I20260926 06:28:15.045004 18150 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:33235
I20260926 06:28:15.045113 18299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:33235 every 8 connection(s)
I20260926 06:28:15.045445 18150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:28:15.047842 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 18150
I20260926 06:28:15.048240 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:36829
--local_ip_for_outbound_sockets=127.2.39.194
--tserver_master_addrs=127.2.39.254:42591
--webserver_port=42061
--webserver_interface=127.2.39.194
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:28:15.050658 18192 log.cc:824] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Log is configured to *not* fsync() on all Append() calls
Sep 26 06:28:15 dist-test-slave-fkbn krb5kdc[17531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404094, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:15.055763 18192 tablet_bootstrap.cc:492] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:28:15.056115 18192 tablet_bootstrap.cc:492] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Bootstrap complete.
I20260926 06:28:15.057246 18192 ts_tablet_manager.cc:1397] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.012s
I20260926 06:28:15.058418 18192 raft_consensus.cc:348] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:28:15.058727 18192 raft_consensus.cc:729] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90cf626a84c644dfba730fa44d69e3e7, State: Initialized, Role: FOLLOWER
I20260926 06:28:15.058867 18192 consensus_queue.cc:260] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } }
I20260926 06:28:15.059049 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.046206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:38365)
0926 06:28:15.046465 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:15.046517 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:15.047025 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:15.047483 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:15.047486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:15.047768 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:15.048037 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.048042 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.049358 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.049360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.050560 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.050564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.051466 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.051467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.051577 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.052160 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:15.052168 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:15.054160 (+ 1992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:15.056780 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:15.056789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:15.056791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:15.057170 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:15.057658 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:15.057659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:15.057660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:15.057793 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:15.058382 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:15.058387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:15.058529 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 06:28:15.058646 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 06:28:15.058808 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":143,"threads_started":1}
I20260926 06:28:15.059129 18192 ts_tablet_manager.cc:1428] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Time spent starting tablet: real 0.002s user 0.004s sys 0.002s
I20260926 06:28:15.059763 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.046347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:38365 (local address 127.2.39.254:42591)
0926 06:28:15.046537 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.046545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.046619 (+ 74us) server_negotiation.cc:407] Connection header received
0926 06:28:15.047249 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.047254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.047295 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.047444 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:15.048153 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.049209 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.050676 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.051335 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.051458 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.054292 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:15.054296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:15.054296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:15.054308 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:15.056683 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:15.057371 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:15.057377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:15.057378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:15.057520 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:15.057882 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:15.057884 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:15.057884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:15.058250 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:15.058382 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.058891 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.059495 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260926 06:28:15.060384 18300 heartbeater.cc:347] Connected to a master server at 127.2.39.254:42591
I20260926 06:28:15.060530 18300 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:15.060747 18300 heartbeater.cc:511] Master 127.2.39.254:42591 requested a full tablet report, sending...
I20260926 06:28:15.061550 18112 ts_manager.cc:194] Registered new tserver with Master: 90cf626a84c644dfba730fa44d69e3e7 (127.2.39.193:33235)
I20260926 06:28:15.062568 18112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:38365
I20260926 06:28:15.063582 18300 heartbeater.cc:503] Master 127.2.39.254:42591 was elected leader, sending a full tablet report...
I20260926 06:28:15.070155 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.062181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:33235 (local address 127.0.0.1:51508)
0926 06:28:15.062345 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:15.062385 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:15.062445 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:15.067411 (+ 4966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:15.067414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:15.067418 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:15.067488 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.067490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.068343 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.068344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.069127 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.069130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.069835 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.069836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.069944 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.069953 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 06:28:15.070037 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 06:28:15.070089 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":97,"threads_started":1}
I20260926 06:28:15.070152 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.062252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.2.39.193:33235)
0926 06:28:15.067255 (+ 5003us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.067259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.067268 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:28:15.067294 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.067295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.067306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.067380 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:15.067590 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.068262 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.069230 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.069737 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.069835 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.069888 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.070054 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.070089 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4975,"thread_start_us":84,"threads_started":1}
I20260926 06:28:15.070502 18234 tablet_service.cc:1514] Processing DeleteTablet for tablet f81457c0de574b52be08f262becd9b40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:27:54 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:51508
I20260926 06:28:15.070962 18309 tablet_replica.cc:330] stopping tablet replica
I20260926 06:28:15.071075 18309 raft_consensus.cc:2229] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:15.071147 18309 raft_consensus.cc:2258] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:15.071491 18309 ts_tablet_manager.cc:1905] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:15.074162 18309 ts_tablet_manager.cc:1918] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:15.074249 18309 log.cc:1196] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal/wals/f81457c0de574b52be08f262becd9b40
I20260926 06:28:15.074548 18309 ts_tablet_manager.cc:1939] T f81457c0de574b52be08f262becd9b40 P 90cf626a84c644dfba730fa44d69e3e7: Deleting consensus metadata
I20260926 06:28:15.075116 18097 catalog_manager.cc:4841] TS 90cf626a84c644dfba730fa44d69e3e7 (127.2.39.193:33235): tablet f81457c0de574b52be08f262becd9b40 (table test-table [id=3f9233b6cd66417dbcc77c5414bf7417]) successfully deleted
W20260926 06:28:15.144474 18304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:15.144765 18304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:15.144812 18304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:15.146274 18304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:15.146330 18304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:15.146363 18304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:28:15.147886 18304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:36829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=42061
--tserver_master_addrs=127.2.39.254:42591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:15.148078 18304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:15.148262 18304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:15.153205 18317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:15.153210 18314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:15.153232 18315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:15.153578 18304 server_base.cc:1034] running on GCE node
I20260926 06:28:15.153774 18304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:15.154026 18304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:15.155192 18304 hybrid_clock.cc:648] HybridClock initialized: now 1790404095155173 us; error 36 us; skew 500 ppm
Sep 26 06:28:15 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:15.157521 18304 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:28:15.157940 18304 webserver.cc:460] Webserver started at http://127.2.39.194:42061/ using document root <none> and password file <none>
I20260926 06:28:15.158138 18304 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:15.158188 18304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:15.159440 18304 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:28:15.160012 18324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:15.160197 18304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:28:15.160274 18304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "9c969493567644d39e8a4bc024eb1bfc"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "e6eff7bbb68ed83b43607a0a4d80e6bc"
server_key_iv: "e056660f9d456f3ba8a6ba77e3717b45"
server_key_version: "encryptionkey@0"
I20260926 06:28:15.160636 18304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:15.176465 18304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:15.178483 18304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:15.178601 18304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:15.178776 18304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:15.179075 18304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:15.179108 18304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:15.179164 18304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:15.179188 18304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:15.185027 18304 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:36829
I20260926 06:28:15.185112 18437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:36829 every 8 connection(s)
I20260926 06:28:15.185441 18304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
Sep 26 06:28:15 dist-test-slave-fkbn krb5kdc[17531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:15.194527 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.186267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:49197 (local address 127.2.39.254:42591)
0926 06:28:15.186316 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.186319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.186525 (+ 206us) server_negotiation.cc:407] Connection header received
0926 06:28:15.186986 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.186987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.186996 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.187053 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:15.187921 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.188409 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.189352 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.189609 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.189697 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.191842 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:15.191843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:15.191843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:15.191852 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:15.193109 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:15.193561 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:15.193562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:15.193563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:15.193647 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:15.193892 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:15.193893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:15.193893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:15.194014 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:15.194087 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.194345 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.194435 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:28:15.194523 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 18304
I20260926 06:28:15.194726 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.186232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:49197)
0926 06:28:15.186395 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:15.186437 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:15.186903 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:15.187105 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:15.187108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:15.187655 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:15.187842 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.187847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.188488 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.188488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.189275 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.189277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.189705 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.189705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.189802 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.190330 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:15.190338 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:15.191743 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:15.193224 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:15.193231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:15.193233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:15.193467 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:15.193720 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:15.193721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:15.193722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:15.193823 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:15.194090 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:15.194097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:15.194219 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 06:28:15.194324 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 06:28:15.194488 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260926 06:28:15.194845 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:43493
--local_ip_for_outbound_sockets=127.2.39.195
--tserver_master_addrs=127.2.39.254:42591
--webserver_port=35387
--webserver_interface=127.2.39.195
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:28:15.195495 18438 heartbeater.cc:347] Connected to a master server at 127.2.39.254:42591
I20260926 06:28:15.195598 18438 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:15.195771 18438 heartbeater.cc:511] Master 127.2.39.254:42591 requested a full tablet report, sending...
I20260926 06:28:15.196188 18112 ts_manager.cc:194] Registered new tserver with Master: 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829)
I20260926 06:28:15.196594 18112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:49197
W20260926 06:28:15.274351 18441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:15.274547 18441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:15.274569 18441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:15.275707 18441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:15.275749 18441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:15.275775 18441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:28:15.277168 18441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:43493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=35387
--tserver_master_addrs=127.2.39.254:42591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:15.277401 18441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:15.277647 18441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:15.282778 18449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:15.282773 18446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:15.282794 18447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:15.282932 18441 server_base.cc:1034] running on GCE node
I20260926 06:28:15.283196 18441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:15.283456 18441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:15.284626 18441 hybrid_clock.cc:648] HybridClock initialized: now 1790404095284613 us; error 43 us; skew 500 ppm
Sep 26 06:28:15 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:15.287104 18441 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:28:15.287535 18441 webserver.cc:460] Webserver started at http://127.2.39.195:35387/ using document root <none> and password file <none>
I20260926 06:28:15.287766 18441 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:15.287855 18441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:15.289121 18441 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:28:15.289776 18456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:15.289934 18441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:28:15.290030 18441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "41f3cab812394cb68e97440801aaea62"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "160ef42dd51a91eb0e27e319c3d2317a"
server_key_iv: "63d2382e96a1fd5bad5c43532572106c"
server_key_version: "encryptionkey@0"
I20260926 06:28:15.290388 18441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:15.312417 18441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:15.314313 18441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:15.314477 18441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:15.314707 18441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:15.315016 18441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:15.315090 18441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:15.315137 18441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:15.315162 18441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:15.320930 18441 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:43493
I20260926 06:28:15.320971 18569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:43493 every 8 connection(s)
I20260926 06:28:15.321288 18441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:28:15 dist-test-slave-fkbn krb5kdc[17531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404095, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:15.329749 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.322038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:33981 (local address 127.2.39.254:42591)
0926 06:28:15.322082 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.322084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.322286 (+ 202us) server_negotiation.cc:407] Connection header received
0926 06:28:15.322748 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.322749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.322759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.322805 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:15.323671 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.324172 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.325124 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.325315 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.325381 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.327402 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:15.327403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:15.327404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:15.327409 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:15.328592 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:15.328954 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:15.328955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:15.328955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:15.329004 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:15.329209 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:15.329209 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:15.329210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:15.329305 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:15.329364 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.329618 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.329688 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:28:15.329905 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.322036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:33981)
0926 06:28:15.322188 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:15.322222 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:15.322668 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:15.322865 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:15.322868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:15.323407 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:15.323587 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.323591 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.324262 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.324262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.325047 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.325049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.325390 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.325391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.325486 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.326025 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:15.326034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:15.327329 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:15.328661 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:15.328666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:15.328668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:15.328889 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:15.329057 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:15.329058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:15.329058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:15.329151 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:15.329363 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:15.329366 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:15.329468 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 06:28:15.329548 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 06:28:15.329716 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260926 06:28:15.330157 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 18441
I20260926 06:28:15.330802 18570 heartbeater.cc:347] Connected to a master server at 127.2.39.254:42591
I20260926 06:28:15.330909 18570 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:15.331097 18570 heartbeater.cc:511] Master 127.2.39.254:42591 requested a full tablet report, sending...
I20260926 06:28:15.331511 18112 ts_manager.cc:194] Registered new tserver with Master: 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493)
I20260926 06:28:15.331871 18112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:33981
I20260926 06:28:15.336525 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.330585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.2.39.254:42591)
0926 06:28:15.330645 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.330647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.330798 (+ 151us) server_negotiation.cc:407] Connection header received
0926 06:28:15.330828 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.330829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.330839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.330904 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:15.331569 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.332097 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.332859 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.333058 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.333139 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.334232 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:15.334233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:15.334234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:15.334240 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:15.335417 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:15.335780 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:15.335781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:15.335781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:15.335833 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:15.336044 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:15.336044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:15.336045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:15.336145 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:15.336316 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.336362 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.336432 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:28:15.337205 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:28:15.339553 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.337514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52604 (local address 127.2.39.254:42591)
0926 06:28:15.337593 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.337595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.337631 (+ 36us) server_negotiation.cc:407] Connection header received
0926 06:28:15.337698 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.337698 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.337726 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.337783 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:28:15.337853 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.338270 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.338980 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.339147 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.339204 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.339342 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:28:15.339377 (+ 35us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 06:28:15.339481 (+ 104us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52604: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 06:28:15.339607 18303 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52604: token expired
W20260926 06:28:15.339680 18039 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.39.254:42591)
I20260926 06:28:15.344987 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.339924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.2.39.254:42591)
0926 06:28:15.339975 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.339976 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.339985 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:28:15.340011 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.340012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.340023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.340064 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:15.340512 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.340923 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.341681 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.341844 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.341895 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.342858 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:15.342858 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:15.342859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:15.342864 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:15.344123 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:15.344404 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:15.344405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:15.344405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:15.344449 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:15.344605 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:15.344620 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:15.344620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:15.344716 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:15.344778 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.344869 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.344939 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 06:28:15.346118 18112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:28:15.347657 18112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:28:15.350003 18234 tablet_service.cc:1467] Processing CreateTablet for tablet 968956b20502409aa66811113af5ac75 (DEFAULT_TABLE table=test-table [id=022bde4ba76e4b45a4d3c7e1e0fa0c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:15.350207 18234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968956b20502409aa66811113af5ac75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:15.352185 18192 tablet_bootstrap.cc:492] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Bootstrap starting.
I20260926 06:28:15.352507 18192 tablet_bootstrap.cc:654] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:15.353171 18192 tablet_bootstrap.cc:492] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: No bootstrap required, opened a new log
I20260926 06:28:15.353226 18192 ts_tablet_manager.cc:1397] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 06:28:15.353336 18192 raft_consensus.cc:348] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:15.353423 18192 raft_consensus.cc:374] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:15.353464 18192 raft_consensus.cc:729] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90cf626a84c644dfba730fa44d69e3e7, State: Initialized, Role: FOLLOWER
I20260926 06:28:15.353507 18192 consensus_queue.cc:260] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:15.353624 18192 ts_tablet_manager.cc:1428] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 06:28:15.353695 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.349593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:43493 (local address 127.0.0.1:35014)
0926 06:28:15.349853 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:15.349863 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:15.349903 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:15.350236 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:15.350237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:15.350239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:15.350303 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.350304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.351685 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.351686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.352464 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.352467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.353428 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.353429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.353490 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.353493 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:15.353554 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:28:15.353630 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242}
I20260926 06:28:15.353868 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.349873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.2.39.195:43493)
0926 06:28:15.350056 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.350061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.350070 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:28:15.350106 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.350108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.350122 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.350231 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:15.350499 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.351520 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.352547 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.353318 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.353423 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.353485 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.353732 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.353778 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":141,"threads_started":1}
I20260926 06:28:15.354589 18504 tablet_service.cc:1467] Processing CreateTablet for tablet 968956b20502409aa66811113af5ac75 (DEFAULT_TABLE table=test-table [id=022bde4ba76e4b45a4d3c7e1e0fa0c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:15.354880 18504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968956b20502409aa66811113af5ac75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:15.356055 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.349690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:36829 (local address 127.0.0.1:53186)
0926 06:28:15.352229 (+ 2539us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:15.352240 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:15.352401 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:15.352767 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:15.352768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:15.352770 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:15.352813 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.352814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.354065 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.354066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.354952 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.354955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.355887 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.355887 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.355932 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.355935 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:15.355975 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 06:28:15.356008 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2527,"thread_start_us":58,"threads_started":1}
I20260926 06:28:15.356113 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.350507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.2.39.194:36829)
0926 06:28:15.351011 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.351015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.352374 (+ 1359us) server_negotiation.cc:407] Connection header received
0926 06:28:15.352503 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.352506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.352523 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.352639 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:15.352897 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.353957 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.355046 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.355794 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.355886 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.355940 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.355996 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.356039 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":121,"threads_started":1}
I20260926 06:28:15.356644 18372 tablet_service.cc:1467] Processing CreateTablet for tablet 968956b20502409aa66811113af5ac75 (DEFAULT_TABLE table=test-table [id=022bde4ba76e4b45a4d3c7e1e0fa0c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:15.356930 18372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 968956b20502409aa66811113af5ac75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:15.357825 18579 tablet_bootstrap.cc:492] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: Bootstrap starting.
I20260926 06:28:15.358448 18579 tablet_bootstrap.cc:654] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:15.358959 18579 log.cc:824] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:15.359616 18579 tablet_bootstrap.cc:492] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: No bootstrap required, opened a new log
I20260926 06:28:15.359691 18579 ts_tablet_manager.cc:1397] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:28:15.359839 18581 tablet_bootstrap.cc:492] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: Bootstrap starting.
I20260926 06:28:15.360436 18581 tablet_bootstrap.cc:654] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:15.360984 18581 log.cc:824] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:15.361940 18581 tablet_bootstrap.cc:492] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: No bootstrap required, opened a new log
I20260926 06:28:15.361861 18579 raft_consensus.cc:348] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:15.362005 18581 ts_tablet_manager.cc:1397] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:28:15.362017 18579 raft_consensus.cc:374] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:15.362066 18579 raft_consensus.cc:729] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f3cab812394cb68e97440801aaea62, State: Initialized, Role: FOLLOWER
I20260926 06:28:15.362185 18579 consensus_queue.cc:260] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:15.362449 18579 ts_tablet_manager.cc:1428] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:28:15.362469 18570 heartbeater.cc:503] Master 127.2.39.254:42591 was elected leader, sending a full tablet report...
I20260926 06:28:15.363864 18581 raft_consensus.cc:348] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:15.364044 18581 raft_consensus.cc:374] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:15.364084 18581 raft_consensus.cc:729] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c969493567644d39e8a4bc024eb1bfc, State: Initialized, Role: FOLLOWER
I20260926 06:28:15.364190 18581 consensus_queue.cc:260] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:15.364452 18581 ts_tablet_manager.cc:1428] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:28:15.364475 18438 heartbeater.cc:503] Master 127.2.39.254:42591 was elected leader, sending a full tablet report...
W20260926 06:28:15.436055 18439 tablet.cc:2369] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:28:15.546195 18301 tablet.cc:2369] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:28:15.571846 18571 tablet.cc:2369] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:28:15.766726 18306 raft_consensus.cc:482] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:15.766903 18306 raft_consensus.cc:504] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:15.767256 18306 leader_election.cc:288] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493), 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829)
I20260926 06:28:15.770583 18586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.767733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:36829 (local address 127.2.39.193:60537)
0926 06:28:15.767809 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:15.767819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:15.767875 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:15.768060 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:15.768060 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:15.768066 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:15.768149 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.768151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.768945 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.768945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.769700 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.769702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.770331 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.770333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.770396 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.770401 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 06:28:15.770452 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:28:15.770491 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":14,"thread_start_us":38,"threads_started":1}
I20260926 06:28:15.770602 18577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.767859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:60537 (local address 127.2.39.194:36829)
0926 06:28:15.767910 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.767912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.767921 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:28:15.767949 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.767950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.767962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.768026 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:15.768221 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.768860 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.769791 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.770244 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.770305 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.770312 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.770479 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.770524 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 06:28:15.770692 18576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.767596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:57593 (local address 127.2.39.195:43493)
0926 06:28:15.767675 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.767677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.767788 (+ 111us) server_negotiation.cc:407] Connection header received
0926 06:28:15.767894 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.767896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.767906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.767960 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:15.768112 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.768697 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.769928 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.770310 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.770381 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.770389 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.770579 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.770622 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:28:15.770735 18585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.767515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:43493 (local address 127.2.39.193:57593)
0926 06:28:15.767682 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:15.767695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:15.767811 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:15.767965 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:15.767966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:15.767969 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:15.768036 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.768038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.768783 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.768784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.769837 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:15.769840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.770409 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.770410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:15.770494 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.770498 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:15.770559 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:28:15.770601 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260926 06:28:15.771150 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968956b20502409aa66811113af5ac75" candidate_uuid: "90cf626a84c644dfba730fa44d69e3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c969493567644d39e8a4bc024eb1bfc" is_pre_election: true
I20260926 06:28:15.771186 18524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968956b20502409aa66811113af5ac75" candidate_uuid: "90cf626a84c644dfba730fa44d69e3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f3cab812394cb68e97440801aaea62" is_pre_election: true
I20260926 06:28:15.771313 18392 raft_consensus.cc:2454] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90cf626a84c644dfba730fa44d69e3e7 in term 0.
I20260926 06:28:15.771342 18524 raft_consensus.cc:2454] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90cf626a84c644dfba730fa44d69e3e7 in term 0.
I20260926 06:28:15.771541 18189 leader_election.cc:302] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41f3cab812394cb68e97440801aaea62, 90cf626a84c644dfba730fa44d69e3e7; no voters:
I20260926 06:28:15.771704 18306 raft_consensus.cc:2781] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:28:15.771786 18306 raft_consensus.cc:482] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:28:15.771806 18306 raft_consensus.cc:3037] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:15.772557 18306 raft_consensus.cc:504] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:15.772703 18306 leader_election.cc:288] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 1 election: Requested vote from peers 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493), 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829)
I20260926 06:28:15.772881 18524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968956b20502409aa66811113af5ac75" candidate_uuid: "90cf626a84c644dfba730fa44d69e3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41f3cab812394cb68e97440801aaea62"
I20260926 06:28:15.772962 18524 raft_consensus.cc:3037] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:15.772969 18392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "968956b20502409aa66811113af5ac75" candidate_uuid: "90cf626a84c644dfba730fa44d69e3e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c969493567644d39e8a4bc024eb1bfc"
I20260926 06:28:15.773048 18392 raft_consensus.cc:3037] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:15.773825 18392 raft_consensus.cc:2454] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90cf626a84c644dfba730fa44d69e3e7 in term 1.
I20260926 06:28:15.773826 18524 raft_consensus.cc:2454] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90cf626a84c644dfba730fa44d69e3e7 in term 1.
I20260926 06:28:15.773984 18189 leader_election.cc:302] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41f3cab812394cb68e97440801aaea62, 90cf626a84c644dfba730fa44d69e3e7; no voters:
I20260926 06:28:15.774096 18306 raft_consensus.cc:2781] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:15.774207 18306 raft_consensus.cc:686] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 LEADER]: Becoming Leader. State: Replica: 90cf626a84c644dfba730fa44d69e3e7, State: Running, Role: LEADER
I20260926 06:28:15.774262 18306 consensus_queue.cc:237] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:15.775063 18112 catalog_manager.cc:5495] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90cf626a84c644dfba730fa44d69e3e7 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "90cf626a84c644dfba730fa44d69e3e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } health_report { overall_health: HEALTHY } } }
I20260926 06:28:15.793278 18589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.790221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51516 (local address 127.2.39.193:33235)
0926 06:28:15.790399 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:15.790401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:15.790407 (+ 6us) server_negotiation.cc:407] Connection header received
0926 06:28:15.790440 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:15.790440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:15.790457 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:15.790524 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:28:15.790625 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.791597 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.792392 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:15.792587 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:15.792664 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:15.792851 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:28:15.792956 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:28:15.793056 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:15.793104 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 06:28:15.793123 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260926 06:28:15.794930 18524 raft_consensus.cc:1261] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62 [term 1 FOLLOWER]: Refusing update from remote peer 90cf626a84c644dfba730fa44d69e3e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:15.794971 18392 raft_consensus.cc:1261] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc [term 1 FOLLOWER]: Refusing update from remote peer 90cf626a84c644dfba730fa44d69e3e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:15.795281 18306 consensus_queue.cc:1035] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:15.795439 18587 consensus_queue.cc:1035] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:15.798173 18591 mvcc.cc:205] Tried to move back new op lower bound from 7333495176373088256 to 7333495176292016128. Current Snapshot: MvccSnapshot[applied={T|T < 7333495176373088256}]
I20260926 06:28:15.798782 18592 mvcc.cc:205] Tried to move back new op lower bound from 7333495176373088256 to 7333495176292016128. Current Snapshot: MvccSnapshot[applied={T|T < 7333495176373088256 or (T in {7333495176373088256})}]
I20260926 06:28:15.800125 18111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:15.800227 18111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:15.801010 18111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Sending DeleteTablet for 3 replicas of tablet 968956b20502409aa66811113af5ac75
I20260926 06:28:15.801318 18504 tablet_service.cc:1514] Processing DeleteTablet for tablet 968956b20502409aa66811113af5ac75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:15 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:35014
I20260926 06:28:15.801400 18234 tablet_service.cc:1514] Processing DeleteTablet for tablet 968956b20502409aa66811113af5ac75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:15 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:51508
I20260926 06:28:15.801448 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 18150
I20260926 06:28:15.801534 18372 tablet_service.cc:1514] Processing DeleteTablet for tablet 968956b20502409aa66811113af5ac75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:15 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:53186
I20260926 06:28:15.802224 18600 tablet_replica.cc:330] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: stopping tablet replica
I20260926 06:28:15.802255 18602 tablet_replica.cc:330] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: stopping tablet replica
I20260926 06:28:15.802330 18600 raft_consensus.cc:2229] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:15.802392 18602 raft_consensus.cc:2229] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:15.802482 18600 raft_consensus.cc:2258] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:15.802512 18602 raft_consensus.cc:2258] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:15.802961 18602 ts_tablet_manager.cc:1905] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:15.802989 18600 ts_tablet_manager.cc:1905] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:15.804723 18602 ts_tablet_manager.cc:1918] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:15.804839 18602 log.cc:1196] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/968956b20502409aa66811113af5ac75
I20260926 06:28:15.804937 18600 ts_tablet_manager.cc:1918] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:15.805007 18600 log.cc:1196] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/968956b20502409aa66811113af5ac75
I20260926 06:28:15.805159 18602 ts_tablet_manager.cc:1939] T 968956b20502409aa66811113af5ac75 P 9c969493567644d39e8a4bc024eb1bfc: Deleting consensus metadata
I20260926 06:28:15.805248 18600 ts_tablet_manager.cc:1939] T 968956b20502409aa66811113af5ac75 P 41f3cab812394cb68e97440801aaea62: Deleting consensus metadata
I20260926 06:28:15.805646 18099 catalog_manager.cc:4841] TS 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829): tablet 968956b20502409aa66811113af5ac75 (table test-table [id=022bde4ba76e4b45a4d3c7e1e0fa0c31]) successfully deleted
I20260926 06:28:15.806047 18099 catalog_manager.cc:4841] TS 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493): tablet 968956b20502409aa66811113af5ac75 (table test-table [id=022bde4ba76e4b45a4d3c7e1e0fa0c31]) successfully deleted
W20260926 06:28:15.808478 18097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:33235 (error 108)
I20260926 06:28:15.808753 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 18304
I20260926 06:28:15.809374 18575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:15.808830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:33235 (local address 127.0.0.1:51528)
0926 06:28:15.809298 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:15.809339 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:33235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430}
W20260926 06:28:15.809494 18097 catalog_manager.cc:4584] TS 90cf626a84c644dfba730fa44d69e3e7 (127.2.39.193:33235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 968956b20502409aa66811113af5ac75: Network error: Client connection negotiation failed: client connection to 127.2.39.193:33235: connect: Connection refused (error 111)
I20260926 06:28:15.814025 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 18441
I20260926 06:28:15.818502 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 18079
I20260926 06:28:35.823987 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:42591
--webserver_interface=127.2.39.254
--webserver_port=34285
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:42591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:35.904983 18604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:35.905217 18604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:35.905254 18604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:35.906437 18604 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 06:28:35.906502 18604 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 06:28:35.906527 18604 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 06:28:35.906556 18604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:28:35.906580 18604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:35.906613 18604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:35.906639 18604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:28:35.906663 18604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:28:35.908095 18604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:42591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:42591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=34285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:35.908355 18604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:35.908574 18604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:35.913236 18610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:35.913430 18612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:35.913540 18609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:35.913540 18604 server_base.cc:1034] running on GCE node
I20260926 06:28:35.913834 18604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:35.914095 18604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:35.915234 18604 hybrid_clock.cc:648] HybridClock initialized: now 1790404115915222 us; error 28 us; skew 500 ppm
Sep 26 06:28:35 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404115, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:35.922662 18604 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:28:35.923156 18604 webserver.cc:460] Webserver started at http://127.2.39.254:34285/ using document root <none> and password file <none>
I20260926 06:28:35.923408 18604 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:35.923477 18604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:35.924788 18604 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:28:35.925506 18619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:35.925745 18604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:28:35.925858 18604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "4e89906280544dea8e1a75f174ee8102"
format_stamp: "Formatted at 2026-09-26 06:27:53 on dist-test-slave-fkbn"
server_key: "29495ab770a1c00fb6cb10156e38748b"
server_key_iv: "3e2a0d19328238b03ad8d689bb95b25c"
server_key_version: "encryptionkey@0"
I20260926 06:28:35.926175 18604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:35.947978 18604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:35.950253 18604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:35.950387 18604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:35.954909 18604 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:42591
I20260926 06:28:35.954964 18671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:42591 every 8 connection(s)
I20260926 06:28:35.955422 18604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:28:35.956903 18672 sys_catalog.cc:263] Verifying existing consensus state
I20260926 06:28:35.957653 18672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Bootstrap starting.
I20260926 06:28:35.959066 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 18604
I20260926 06:28:35.959365 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:33235
--local_ip_for_outbound_sockets=127.2.39.193
--tserver_master_addrs=127.2.39.254:42591
--webserver_port=35585
--webserver_interface=127.2.39.193
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:28:35.966993 18672 log.cc:824] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:35.969703 18672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:28:35.970057 18672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Bootstrap complete.
I20260926 06:28:35.972504 18672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:28:35.972769 18672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e89906280544dea8e1a75f174ee8102, State: Initialized, Role: FOLLOWER
I20260926 06:28:35.972980 18672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:28:35.973081 18672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:28:35.973140 18672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:28:35.973189 18672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 2 FOLLOWER]: Advancing to term 3
I20260926 06:28:35.974511 18672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:28:35.974610 18672 leader_election.cc:302] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e89906280544dea8e1a75f174ee8102; no voters:
I20260926 06:28:35.974833 18672 leader_election.cc:288] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 06:28:35.974880 18676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 06:28:35.975098 18676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [term 3 LEADER]: Becoming Leader. State: Replica: 4e89906280544dea8e1a75f174ee8102, State: Running, Role: LEADER
I20260926 06:28:35.975142 18672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:28:35.975198 18676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } }
I20260926 06:28:35.975500 18676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e89906280544dea8e1a75f174ee8102. Latest consensus state: current_term: 3 leader_uuid: "4e89906280544dea8e1a75f174ee8102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } } }
I20260926 06:28:35.975503 18677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4e89906280544dea8e1a75f174ee8102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e89906280544dea8e1a75f174ee8102" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42591 } } }
I20260926 06:28:35.975687 18677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:35.976112 18676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:35.976280 18683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:28:35.977115 18683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:28:35.977221 18683 catalog_manager.cc:1259] Loaded cluster ID: 7cfdf676bb56481383e513f18bc4a9ba
I20260926 06:28:35.977310 18683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:28:35.978672 18683 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:28:35.978850 18683 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Loaded TSK: 0
I20260926 06:28:35.978892 18683 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4e89906280544dea8e1a75f174ee8102: Loaded TSK: 1
F20260926 06:28:35.979125 18683 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff963eb79ad google::LogMessage::Fail() at ??:0
@ 0x7ff963eba88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff963eb74de google::LogMessage::Flush() at ??:0
@ 0x7ff963eb8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff966624104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff966621d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff96be111cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff96be1148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff96be11594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff96bdd7c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff96bde1ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff965d850d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff965d7a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff966bde6db start_thread at ??:0
@ 0x7ff9629b561f clone at ??:0
W20260926 06:28:36.055586 18674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:36.055816 18674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:36.055853 18674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:36.056993 18674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:36.057060 18674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:36.057104 18674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:28:36.058944 18674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:33235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=35585
--tserver_master_addrs=127.2.39.254:42591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:36.059244 18674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:36.059515 18674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:36.065750 18713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:36.065743 18714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:36.065766 18716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:36.066075 18674 server_base.cc:1034] running on GCE node
I20260926 06:28:36.066320 18674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:36.066584 18674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:36.067756 18674 hybrid_clock.cc:648] HybridClock initialized: now 1790404116067729 us; error 40 us; skew 500 ppm
Sep 26 06:28:36 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404116, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:36.070716 18674 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:28:36.071179 18674 webserver.cc:460] Webserver started at http://127.2.39.193:35585/ using document root <none> and password file <none>
I20260926 06:28:36.071437 18674 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:36.071540 18674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:36.072902 18674 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:28:36.073671 18723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:36.073865 18674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 06:28:36.073983 18674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "90cf626a84c644dfba730fa44d69e3e7"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "f3b814d2450a63ae99ded19d649563a6"
server_key_iv: "81698ac66a0d96a59d49d26e3ada284f"
server_key_version: "encryptionkey@0"
I20260926 06:28:36.074369 18674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:36.092504 18674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:36.094971 18674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:36.095160 18674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:36.095399 18674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:36.095844 18730 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 06:28:36.097867 18674 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 06:28:36.097926 18674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 06:28:36.097970 18674 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 06:28:36.098609 18674 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 06:28:36.098668 18674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 06:28:36.098770 18730 tablet_bootstrap.cc:492] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Bootstrap starting.
I20260926 06:28:36.106072 18674 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:33235
I20260926 06:28:36.106112 18837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:33235 every 8 connection(s)
I20260926 06:28:36.106483 18674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:28:36.110034 18730 log.cc:824] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:36.112720 18730 tablet_bootstrap.cc:492] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 06:28:36.113067 18730 tablet_bootstrap.cc:492] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Bootstrap complete.
I20260926 06:28:36.114343 18730 ts_tablet_manager.cc:1397] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260926 06:28:36.115103 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 18674
I20260926 06:28:36.115378 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:36829
--local_ip_for_outbound_sockets=127.2.39.194
--tserver_master_addrs=127.2.39.254:42591
--webserver_port=42061
--webserver_interface=127.2.39.194
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:28:36.115768 18730 raft_consensus.cc:348] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:36.116106 18730 raft_consensus.cc:729] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90cf626a84c644dfba730fa44d69e3e7, State: Initialized, Role: FOLLOWER
I20260926 06:28:36.116267 18730 consensus_queue.cc:260] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:36.116544 18730 ts_tablet_manager.cc:1428] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260926 06:28:36.183871 18840 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:28:36.107198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:51287)
0926 06:28:36.107394 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:36.107429 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:36.107883 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:36.183652 (+ 75769us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
W20260926 06:28:36.184217 18838 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:42591 (0 consecutive failures): Network error: Failed to ping master at 127.2.39.254:42591: Client connection negotiation failed: client connection to 127.2.39.254:42591: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 06:28:36.184653 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:36.184525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:60977)
0926 06:28:36.184587 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:36.184633 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 06:28:36.185021 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:36.184921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:52711)
0926 06:28:36.184976 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:36.184999 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 06:28:36.185173 18838 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:28:36.198179 18843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:36.198406 18843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:36.198447 18843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:36.199874 18843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:36.199926 18843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:36.199954 18843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:28:36.201396 18843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:36829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=42061
--tserver_master_addrs=127.2.39.254:42591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:36.201666 18843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:36.201866 18843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:36.206797 18851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:36.206799 18849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:36.206831 18848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:36.207024 18843 server_base.cc:1034] running on GCE node
I20260926 06:28:36.207274 18843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:36.207520 18843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:36.208683 18843 hybrid_clock.cc:648] HybridClock initialized: now 1790404116208678 us; error 36 us; skew 500 ppm
Sep 26 06:28:36 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404116, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:36.211181 18843 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:28:36.211609 18843 webserver.cc:460] Webserver started at http://127.2.39.194:42061/ using document root <none> and password file <none>
I20260926 06:28:36.211833 18843 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:36.211905 18843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:36.213213 18843 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:28:36.213881 18858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:36.214056 18843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:28:36.214157 18843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "9c969493567644d39e8a4bc024eb1bfc"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "e6eff7bbb68ed83b43607a0a4d80e6bc"
server_key_iv: "e056660f9d456f3ba8a6ba77e3717b45"
server_key_version: "encryptionkey@0"
I20260926 06:28:36.214517 18843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:36.230111 18843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:36.231905 18843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:36.232110 18843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:36.232337 18843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:36.232649 18843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:36.232707 18843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:36.232748 18843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:36.232774 18843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:36.238663 18843 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:36829
I20260926 06:28:36.238772 18971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:36829 every 8 connection(s)
I20260926 06:28:36.239082 18843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:28:36.240103 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:36.239736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:50777)
0926 06:28:36.239892 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:36.239956 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260926 06:28:36.240502 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 18843
W20260926 06:28:36.240527 18972 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:42591 (0 consecutive failures): Network error: Failed to ping master at 127.2.39.254:42591: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
I20260926 06:28:36.240806 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:43493
--local_ip_for_outbound_sockets=127.2.39.195
--tserver_master_addrs=127.2.39.254:42591
--webserver_port=35387
--webserver_interface=127.2.39.195
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:42065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:28:36.240828 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:36.240715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:55251)
0926 06:28:36.240768 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:36.240801 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 06:28:36.241277 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:36.241166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:56971)
0926 06:28:36.241218 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:36.241247 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 06:28:36.241436 18972 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 06:28:36.321236 18975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:36.321432 18975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:36.321455 18975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:36.322634 18975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:36.322676 18975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:36.322736 18975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:28:36.324188 18975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:42065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:43493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=35387
--tserver_master_addrs=127.2.39.254:42591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:36.324429 18975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:36.324658 18975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:36.329660 18983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:36.329655 18981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:36.329711 18975 server_base.cc:1034] running on GCE node
W20260926 06:28:36.329635 18980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:36.330112 18975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:36.330305 18975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:36.331444 18975 hybrid_clock.cc:648] HybridClock initialized: now 1790404116331430 us; error 29 us; skew 500 ppm
Sep 26 06:28:36 dist-test-slave-fkbn krb5kdc[17531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404116, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:36.333837 18975 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:28:36.334280 18975 webserver.cc:460] Webserver started at http://127.2.39.195:35387/ using document root <none> and password file <none>
I20260926 06:28:36.334452 18975 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:36.334496 18975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:36.335742 18975 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:28:36.336272 18990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:36.336462 18975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:28:36.336530 18975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "41f3cab812394cb68e97440801aaea62"
format_stamp: "Formatted at 2026-09-26 06:27:54 on dist-test-slave-fkbn"
server_key: "160ef42dd51a91eb0e27e319c3d2317a"
server_key_iv: "63d2382e96a1fd5bad5c43532572106c"
server_key_version: "encryptionkey@0"
I20260926 06:28:36.336867 18975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:36.355782 18975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:36.358134 18975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:36.358301 18975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:36.358529 18975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:36.358829 18975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:36.358886 18975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:36.358927 18975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:36.358955 18975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:36.364751 18975 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:43493
I20260926 06:28:36.364805 19103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:43493 every 8 connection(s)
I20260926 06:28:36.365116 18975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:28:36.365621 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 18975
I20260926 06:28:36.366222 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:36.365788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:34187)
0926 06:28:36.365945 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:36.366036 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
W20260926 06:28:36.366603 19104 heartbeater.cc:644] Failed to heartbeat to 127.2.39.254:42591 (0 consecutive failures): Network error: Failed to ping master at 127.2.39.254:42591: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
I20260926 06:28:36.366874 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:36.366782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:47011)
0926 06:28:36.366828 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:36.366854 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 06:28:36.367177 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:36.367095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:57191)
0926 06:28:36.367141 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:36.367163 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 06:28:36.367291 19104 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 06:28:37.185786 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:37.185520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:60937)
0926 06:28:37.185695 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:37.185729 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
I20260926 06:28:37.241961 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:37.241776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:58435)
0926 06:28:37.241883 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:37.241916 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
I20260926 06:28:37.367836 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:37.367635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:49737)
0926 06:28:37.367752 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:37.367785 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
I20260926 06:28:37.787104 19111 raft_consensus.cc:482] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:37.787245 19111 raft_consensus.cc:504] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:37.787566 19111 leader_election.cc:288] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493), 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829)
Sep 26 06:28:37 dist-test-slave-fkbn krb5kdc[17531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404116, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.195@KRBTEST.COM
Sep 26 06:28:37 dist-test-slave-fkbn krb5kdc[17531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404116, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.194@KRBTEST.COM
I20260926 06:28:37.798375 19114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:37.787902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:36829 (local address 127.2.39.193:34381)
0926 06:28:37.788056 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:37.788066 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:37.788137 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:37.788815 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:37.788818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:37.789184 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:37.789376 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:37.789381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:37.790531 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:37.790532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:37.791313 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:37.791315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:37.791913 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:37.791914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:37.792022 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:37.792657 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:37.792662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:37.794599 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:37.796734 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:37.796740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:37.796742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:37.796941 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:37.797195 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:37.797196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:37.797196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:37.797303 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:37.797812 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:37.797818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:37.797957 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 06:28:37.798086 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 06:28:37.798277 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":39,"thread_start_us":68,"threads_started":1}
I20260926 06:28:37.798388 19115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:37.788027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:34381 (local address 127.2.39.194:36829)
0926 06:28:37.788142 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:37.788152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:37.788180 (+ 28us) server_negotiation.cc:407] Connection header received
0926 06:28:37.788584 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:37.788586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:37.788626 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:37.788782 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:37.789499 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:37.790399 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:37.791410 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:37.791808 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:37.791942 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:37.794704 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:37.794707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:37.794708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:37.794718 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:37.796657 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:37.797032 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:37.797034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:37.797034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:37.797113 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:37.797384 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:37.797385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:37.797385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:37.797704 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:37.797783 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:37.798152 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 06:28:37.798275 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
I20260926 06:28:37.798460 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:37.787842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:53615 (local address 127.2.39.195:43493)
0926 06:28:37.788045 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:37.788058 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:37.788098 (+ 40us) server_negotiation.cc:407] Connection header received
0926 06:28:37.788586 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:37.788588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:37.788626 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:37.788783 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:37.789525 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:37.790392 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:37.791409 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:37.791808 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:37.791936 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:37.794333 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:37.794337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:37.794337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:37.794348 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:37.796652 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:37.797243 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:37.797245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:37.797245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:37.797335 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:37.797558 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:37.797559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:37.797559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:37.797871 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:37.797949 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:37.798173 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 06:28:37.798339 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260926 06:28:37.798475 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:37.787724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:43493 (local address 127.2.39.193:53615)
0926 06:28:37.787834 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:37.787845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:37.787898 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:37.788815 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:37.788818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:37.789183 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:37.789376 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:37.789382 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:37.790502 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:37.790503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:37.791312 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:37.791314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:37.791909 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:37.791910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:37.792020 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:37.792593 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:37.792602 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:37.794225 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:37.797021 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:37.797022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:37.797022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:37.797145 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:37.797419 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:37.797420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:37.797420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:37.797484 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:37.797942 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:37.797943 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:37.798006 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 06:28:37.798088 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 06:28:37.798356 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":194,"thread_start_us":76,"threads_started":1}
W20260926 06:28:37.799561 18727 leader_election.cc:341] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829): Not found: Tablet not found: 968956b20502409aa66811113af5ac75
W20260926 06:28:37.799638 18727 leader_election.cc:341] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493): Not found: Tablet not found: 968956b20502409aa66811113af5ac75
I20260926 06:28:37.799655 18727 leader_election.cc:302] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90cf626a84c644dfba730fa44d69e3e7; no voters: 41f3cab812394cb68e97440801aaea62, 9c969493567644d39e8a4bc024eb1bfc
I20260926 06:28:37.799752 19111 raft_consensus.cc:2726] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:28:38.186460 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:38.186296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:40203)
0926 06:28:38.186386 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:38.186422 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 06:28:38.242558 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:38.242363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:38139)
0926 06:28:38.242477 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:38.242518 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260926 06:28:38.368425 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:38.368235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:54303)
0926 06:28:38.368359 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:38.368393 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260926 06:28:39.187273 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:39.187005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:43287)
0926 06:28:39.187167 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:39.187212 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260926 06:28:39.243166 19119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:39.242968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:57073)
0926 06:28:39.243086 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:39.243124 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260926 06:28:39.369179 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:39.368937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:34231)
0926 06:28:39.369081 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:39.369137 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260926 06:28:39.387996 19121 raft_consensus.cc:482] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:39.388123 19121 raft_consensus.cc:504] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:39.388311 19121 leader_election.cc:288] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493), 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829)
W20260926 06:28:39.388741 18727 leader_election.cc:341] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493): Not found: Tablet not found: 968956b20502409aa66811113af5ac75
W20260926 06:28:39.388824 18727 leader_election.cc:341] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829): Not found: Tablet not found: 968956b20502409aa66811113af5ac75
I20260926 06:28:39.388842 18727 leader_election.cc:302] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90cf626a84c644dfba730fa44d69e3e7; no voters: 41f3cab812394cb68e97440801aaea62, 9c969493567644d39e8a4bc024eb1bfc
I20260926 06:28:39.388950 19121 raft_consensus.cc:2726] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:28:40.188063 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:40.187843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:41695)
0926 06:28:40.187965 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:40.188016 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260926 06:28:40.243814 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:40.243621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:57005)
0926 06:28:40.243731 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:40.243765 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260926 06:28:40.369875 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:40.369686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:36409)
0926 06:28:40.369807 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:40.369840 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
I20260926 06:28:41.188874 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:41.188612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:40127)
0926 06:28:41.188769 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:41.188816 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
I20260926 06:28:41.244486 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:41.244269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:57819)
0926 06:28:41.244404 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:41.244444 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260926 06:28:41.370556 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:41.370330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:41555)
0926 06:28:41.370477 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:41.370516 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260926 06:28:42.189692 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:42.189401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:59777)
0926 06:28:42.189555 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:42.189643 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260926 06:28:42.245167 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:42.244955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:52247)
0926 06:28:42.245086 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:42.245121 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260926 06:28:42.371206 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:42.371011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:51843)
0926 06:28:42.371136 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:42.371168 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260926 06:28:42.424552 19131 raft_consensus.cc:482] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:42.424646 19131 raft_consensus.cc:504] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:42.424847 19131 leader_election.cc:288] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493), 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829)
W20260926 06:28:42.425258 18727 leader_election.cc:341] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493): Not found: Tablet not found: 968956b20502409aa66811113af5ac75
W20260926 06:28:42.425364 18727 leader_election.cc:341] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829): Not found: Tablet not found: 968956b20502409aa66811113af5ac75
I20260926 06:28:42.425398 18727 leader_election.cc:302] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90cf626a84c644dfba730fa44d69e3e7; no voters: 41f3cab812394cb68e97440801aaea62, 9c969493567644d39e8a4bc024eb1bfc
I20260926 06:28:42.425486 19131 raft_consensus.cc:2726] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:28:43.190424 19132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:43.190179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:59815)
0926 06:28:43.190328 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:43.190376 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260926 06:28:43.245869 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:43.245659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:39781)
0926 06:28:43.245788 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:43.245826 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260926 06:28:43.371827 19134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:43.371613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:49441)
0926 06:28:43.371752 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:43.371788 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260926 06:28:44.191159 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:44.190913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:58325)
0926 06:28:44.191075 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:44.191112 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260926 06:28:44.246527 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:44.246318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:58571)
0926 06:28:44.246435 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:44.246475 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260926 06:28:44.372551 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:44.372352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:38381)
0926 06:28:44.372473 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:44.372511 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260926 06:28:45.191969 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:45.191712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:51997)
0926 06:28:45.191885 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:45.191924 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":80,"threads_started":1}
I20260926 06:28:45.247187 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:45.246975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:33717)
0926 06:28:45.247106 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:45.247144 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260926 06:28:45.373172 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:45.372977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:46099)
0926 06:28:45.373099 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:45.373129 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260926 06:28:46.192616 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:46.192415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:53251)
0926 06:28:46.192537 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:46.192571 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260926 06:28:46.247849 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:46.247606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:38297)
0926 06:28:46.247736 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:46.247804 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":59,"threads_started":1}
I20260926 06:28:46.373843 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:46.373658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:38943)
0926 06:28:46.373770 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:46.373803 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260926 06:28:47.193498 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:47.193249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:38281)
0926 06:28:47.193390 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:47.193439 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260926 06:28:47.248576 19145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:47.248375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:50513)
0926 06:28:47.248495 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:47.248532 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260926 06:28:47.335734 19146 raft_consensus.cc:482] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:47.335827 19146 raft_consensus.cc:504] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41f3cab812394cb68e97440801aaea62" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 43493 } } peers { permanent_uuid: "9c969493567644d39e8a4bc024eb1bfc" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 36829 } } peers { permanent_uuid: "90cf626a84c644dfba730fa44d69e3e7" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 33235 } }
I20260926 06:28:47.336027 19146 leader_election.cc:288] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493), 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829)
W20260926 06:28:47.336488 18727 leader_election.cc:341] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 41f3cab812394cb68e97440801aaea62 (127.2.39.195:43493): Not found: Tablet not found: 968956b20502409aa66811113af5ac75
W20260926 06:28:47.336601 18727 leader_election.cc:341] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9c969493567644d39e8a4bc024eb1bfc (127.2.39.194:36829): Not found: Tablet not found: 968956b20502409aa66811113af5ac75
I20260926 06:28:47.336655 18727 leader_election.cc:302] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 90cf626a84c644dfba730fa44d69e3e7; no voters: 41f3cab812394cb68e97440801aaea62, 9c969493567644d39e8a4bc024eb1bfc
I20260926 06:28:47.336750 19146 raft_consensus.cc:2726] T 968956b20502409aa66811113af5ac75 P 90cf626a84c644dfba730fa44d69e3e7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 06:28:47.374500 19147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:47.374267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:49509)
0926 06:28:47.374387 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:47.374420 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260926 06:28:48.194152 19148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:48.193934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:60059)
0926 06:28:48.194072 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:48.194106 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260926 06:28:48.249289 19149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:48.249061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:60421)
0926 06:28:48.249207 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:48.249244 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260926 06:28:48.375141 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:48.374935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:34317)
0926 06:28:48.375059 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:48.375099 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260926 06:28:49.194823 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:49.194624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:44283)
0926 06:28:49.194741 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:49.194777 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260926 06:28:49.249948 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:49.249753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:58327)
0926 06:28:49.249865 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:49.249900 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
I20260926 06:28:49.375773 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:49.375556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:34921)
0926 06:28:49.375690 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:49.375724 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260926 06:28:50.195519 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:50.195307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:41509)
0926 06:28:50.195436 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:50.195471 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260926 06:28:50.250572 19155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:50.250338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:42899)
0926 06:28:50.250486 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:50.250532 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260926 06:28:50.376406 19156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:50.376179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:56593)
0926 06:28:50.376327 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:50.376363 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260926 06:28:51.196228 19157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:51.196024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.193:49583)
0926 06:28:51.196148 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:51.196184 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260926 06:28:51.251271 19158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:51.251063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.194:57741)
0926 06:28:51.251196 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:51.251231 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260926 06:28:51.370740 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 18674
I20260926 06:28:51.375506 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 18843
I20260926 06:28:51.377238 19159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:51.377050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42591 (local address 127.2.39.195:37691)
0926 06:28:51.377161 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:51.377191 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.254:42591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260926 06:28:51.379840 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 18975
2026-09-26T06:28:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58744 ms)
[----------] 30 tests from SecurityITest (232028 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:28:51 dist-test-slave-fkbn krb5kdc[19162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:28:51 dist-test-slave-fkbn krb5kdc[19162](info): set up 2 sockets
Sep 26 06:28:51 dist-test-slave-fkbn krb5kdc[19162](info): commencing operation
krb5kdc: starting...
W20260926 06:28:52.425276 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:28:52 dist-test-slave-fkbn krb5kdc[19162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:28:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:28:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:28:52.540844 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:34647
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:34647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:52.621469 19175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:52.621754 19175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:52.621796 19175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:52.622933 19175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:28:52.622998 19175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:52.623023 19175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:52.623049 19175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:28:52.623080 19175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:28:52.624457 19175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:34647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:52.624724 19175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:52.624938 19175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:52.630071 19180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:52.630080 19183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:52.630118 19175 server_base.cc:1034] running on GCE node
W20260926 06:28:52.630110 19181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:52.630475 19175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:52.630816 19175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:52.631963 19175 hybrid_clock.cc:648] HybridClock initialized: now 1790404132631953 us; error 30 us; skew 500 ppm
Sep 26 06:28:52 dist-test-slave-fkbn krb5kdc[19162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404132, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:52.634459 19175 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:28:52.634865 19175 webserver.cc:460] Webserver started at http://127.2.39.254:33849/ using document root <none> and password file <none>
I20260926 06:28:52.635095 19175 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:52.635161 19175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:52.635308 19175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:52.636334 19175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946"
format_stamp: "Formatted at 2026-09-26 06:28:52 on dist-test-slave-fkbn"
server_key: "83e049ae671bd4243fed586536109321"
server_key_iv: "7654573f5c8763400775eddb7e41d8cf"
server_key_version: "encryptionkey@0"
I20260926 06:28:52.636684 19175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946"
format_stamp: "Formatted at 2026-09-26 06:28:52 on dist-test-slave-fkbn"
server_key: "83e049ae671bd4243fed586536109321"
server_key_iv: "7654573f5c8763400775eddb7e41d8cf"
server_key_version: "encryptionkey@0"
I20260926 06:28:52.638024 19175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:28:52.638720 19190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:52.638885 19175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:28:52.638978 19175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946"
format_stamp: "Formatted at 2026-09-26 06:28:52 on dist-test-slave-fkbn"
server_key: "83e049ae671bd4243fed586536109321"
server_key_iv: "7654573f5c8763400775eddb7e41d8cf"
server_key_version: "encryptionkey@0"
I20260926 06:28:52.639057 19175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:52.659416 19175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:52.661711 19175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:52.661867 19175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:52.665532 19175 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:34647
I20260926 06:28:52.665611 19242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:34647 every 8 connection(s)
I20260926 06:28:52.665956 19175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:28:52.666855 19243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:52.669147 19243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946: Bootstrap starting.
I20260926 06:28:52.669799 19243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:52.670085 19243 log.cc:824] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:52.670763 19243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946: No bootstrap required, opened a new log
I20260926 06:28:52.672372 19243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34647 } }
I20260926 06:28:52.672508 19243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:52.672564 19243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a5d86b76a8f4bb9835ecef9c4bf7946, State: Initialized, Role: FOLLOWER
I20260926 06:28:52.672708 19243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34647 } }
I20260926 06:28:52.672767 19243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:28:52.672788 19243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:28:52.672868 19243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:52.673418 19243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34647 } }
I20260926 06:28:52.673523 19243 leader_election.cc:302] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a5d86b76a8f4bb9835ecef9c4bf7946; no voters:
I20260926 06:28:52.673727 19243 leader_election.cc:288] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:28:52.673753 19246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:52.673960 19246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [term 1 LEADER]: Becoming Leader. State: Replica: 8a5d86b76a8f4bb9835ecef9c4bf7946, State: Running, Role: LEADER
I20260926 06:28:52.673965 19243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:28:52.674052 19246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34647 } }
I20260926 06:28:52.674415 19247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34647 } } }
I20260926 06:28:52.674499 19247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:52.674557 19248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a5d86b76a8f4bb9835ecef9c4bf7946. Latest consensus state: current_term: 1 leader_uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a5d86b76a8f4bb9835ecef9c4bf7946" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34647 } } }
I20260926 06:28:52.674626 19248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:52.674865 19254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:28:52.675239 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 19175
I20260926 06:28:52.675350 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:28:52.675462 19254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:28:52.675539 2207 external_mini_cluster.cc:1428] Setting key a9ca63844d31fe0e15c7724f1c3ab90b
I20260926 06:28:52.677114 19254 catalog_manager.cc:1347] Generated new cluster ID: 235e00d01bbf4c29abca70869112f1c8
I20260926 06:28:52.677166 19254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:28:52 dist-test-slave-fkbn krb5kdc[19162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:52.687971 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:52.676161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.2.39.254:34647)
0926 06:28:52.676418 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:52.676426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:52.676458 (+ 32us) server_negotiation.cc:407] Connection header received
0926 06:28:52.676572 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:52.676580 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:52.676658 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:52.676824 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:52.677561 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:52.679003 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:52.680102 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:52.680727 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:52.680834 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:52.683952 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:52.683957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:52.683957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:52.683972 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:52.686066 (+ 2094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:52.686451 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:52.686453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:52.686454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:52.686563 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:52.686761 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:52.686762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:52.686762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:52.687057 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:52.687207 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:52.687621 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 06:28:52.687748 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":63,"threads_started":1}
I20260926 06:28:52.697926 19254 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:28:52.698628 19254 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:28:52.707664 19254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946: Generated new TSK 0
I20260926 06:28:52.707837 19254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:28:52.733667 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34647
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:52.812847 19269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:52.813045 19269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:52.813067 19269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:52.814221 19269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:52.814260 19269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:52.814286 19269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:28:52.815697 19269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:52.815925 19269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:52.816152 19269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:52.821112 19277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:52.821221 19275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:52.821121 19274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:52.821376 19269 server_base.cc:1034] running on GCE node
I20260926 06:28:52.821620 19269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:52.821848 19269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:52.822986 19269 hybrid_clock.cc:648] HybridClock initialized: now 1790404132822981 us; error 36 us; skew 500 ppm
Sep 26 06:28:52 dist-test-slave-fkbn krb5kdc[19162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404132, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:52.825443 19269 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:28:52.825898 19269 webserver.cc:460] Webserver started at http://127.2.39.193:45057/ using document root <none> and password file <none>
I20260926 06:28:52.826077 19269 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:52.826118 19269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:52.826251 19269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:52.827251 19269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "569f1c4d975442bc89b83448535a3d54"
format_stamp: "Formatted at 2026-09-26 06:28:52 on dist-test-slave-fkbn"
server_key: "7150accbef404358aed98569feb2d322"
server_key_iv: "fc7febfa848c993641231cabcf1c9f49"
server_key_version: "encryptionkey@0"
I20260926 06:28:52.827615 19269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "569f1c4d975442bc89b83448535a3d54"
format_stamp: "Formatted at 2026-09-26 06:28:52 on dist-test-slave-fkbn"
server_key: "7150accbef404358aed98569feb2d322"
server_key_iv: "fc7febfa848c993641231cabcf1c9f49"
server_key_version: "encryptionkey@0"
I20260926 06:28:52.828876 19269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:28:52.829664 19284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:52.829949 19269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:28:52.830047 19269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "569f1c4d975442bc89b83448535a3d54"
format_stamp: "Formatted at 2026-09-26 06:28:52 on dist-test-slave-fkbn"
server_key: "7150accbef404358aed98569feb2d322"
server_key_iv: "fc7febfa848c993641231cabcf1c9f49"
server_key_version: "encryptionkey@0"
I20260926 06:28:52.830128 19269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:52.850172 19269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:52.852504 19269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:52.852663 19269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:52.852888 19269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:52.853195 19269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:52.853255 19269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:52.853294 19269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:52.853322 19269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:52.858980 19269 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:44995
I20260926 06:28:52.859040 19397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:44995 every 8 connection(s)
I20260926 06:28:52.859325 19269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:28:52 dist-test-slave-fkbn krb5kdc[19162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404132, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:52.867624 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:52.860039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:37515 (local address 127.2.39.254:34647)
0926 06:28:52.860105 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:52.860107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:52.860265 (+ 158us) server_negotiation.cc:407] Connection header received
0926 06:28:52.860717 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:52.860718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:52.860731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:52.860788 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:52.861677 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:52.862173 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:52.863103 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:52.863291 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:52.863340 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:52.865299 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:52.865300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:52.865300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:52.865306 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:52.866551 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:52.866889 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:52.866889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:52.866890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:52.866933 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:52.867149 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:52.867149 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:52.867150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:52.867243 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:52.867291 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:52.867504 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 06:28:52.867564 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:28:52.867797 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:52.859972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:34647 (local address 127.2.39.193:37515)
0926 06:28:52.860137 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:52.860176 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:52.860640 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:52.860838 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:52.860841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:52.861378 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:52.861593 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:52.861598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:52.862249 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:52.862250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:52.863026 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:52.863028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:52.863360 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:52.863361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:52.863456 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:52.863946 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:52.863954 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:52.865225 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:52.866618 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:52.866624 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:52.866625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:52.866827 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:52.867006 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:52.867007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:52.867007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:52.867096 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:52.867296 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:52.867299 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:52.867402 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 06:28:52.867496 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 06:28:52.867607 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260926 06:28:52.868132 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 19269
I20260926 06:28:52.868251 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:28:52.868352 19398 heartbeater.cc:347] Connected to a master server at 127.2.39.254:34647
I20260926 06:28:52.868435 19398 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:52.868489 2207 external_mini_cluster.cc:1428] Setting key 5b7a86e1c56a697284f3af43d498f908
I20260926 06:28:52.868641 19398 heartbeater.cc:511] Master 127.2.39.254:34647 requested a full tablet report, sending...
I20260926 06:28:52.869182 19206 ts_manager.cc:194] Registered new tserver with Master: 569f1c4d975442bc89b83448535a3d54 (127.2.39.193:44995)
I20260926 06:28:52.869874 19206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:37515
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:28:52.892652 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34647
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:52.972836 19403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:52.973038 19403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:52.973061 19403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:52.974385 19403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:52.974437 19403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:52.974464 19403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:28:52.975956 19403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:52.976143 19403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:52.976308 19403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:52.981263 19408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:52.981271 19411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:52.981274 19409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:52.981410 19403 server_base.cc:1034] running on GCE node
I20260926 06:28:52.981745 19403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:52.981931 19403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:52.983063 19403 hybrid_clock.cc:648] HybridClock initialized: now 1790404132983043 us; error 34 us; skew 500 ppm
Sep 26 06:28:52 dist-test-slave-fkbn krb5kdc[19162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404132, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:52.985492 19403 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:28:52.986073 19403 webserver.cc:460] Webserver started at http://127.2.39.194:45737/ using document root <none> and password file <none>
I20260926 06:28:52.986275 19403 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:52.986316 19403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:52.986420 19403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:52.987406 19403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "6153f4cdcedb4641aa4a837924b46a75"
format_stamp: "Formatted at 2026-09-26 06:28:52 on dist-test-slave-fkbn"
server_key: "51a422bc945dd5e52484865942b5b17c"
server_key_iv: "57e1979a0eac4c58ab368e1d99039315"
server_key_version: "encryptionkey@0"
I20260926 06:28:52.987709 19403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "6153f4cdcedb4641aa4a837924b46a75"
format_stamp: "Formatted at 2026-09-26 06:28:52 on dist-test-slave-fkbn"
server_key: "51a422bc945dd5e52484865942b5b17c"
server_key_iv: "57e1979a0eac4c58ab368e1d99039315"
server_key_version: "encryptionkey@0"
I20260926 06:28:52.988940 19403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 06:28:52.989638 19418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:52.989814 19403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:28:52.989905 19403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "6153f4cdcedb4641aa4a837924b46a75"
format_stamp: "Formatted at 2026-09-26 06:28:52 on dist-test-slave-fkbn"
server_key: "51a422bc945dd5e52484865942b5b17c"
server_key_iv: "57e1979a0eac4c58ab368e1d99039315"
server_key_version: "encryptionkey@0"
I20260926 06:28:52.989987 19403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:53.005103 19403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:53.007126 19403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:53.007282 19403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:53.007491 19403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:53.007792 19403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:53.007853 19403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.007893 19403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:53.007920 19403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.013520 19403 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:38029
I20260926 06:28:53.013620 19531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:38029 every 8 connection(s)
I20260926 06:28:53.013932 19403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:28:53.017076 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 19403
I20260926 06:28:53.017223 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:28:53.017421 2207 external_mini_cluster.cc:1428] Setting key 7b8e0896be77ffcf0eaeac73689f9b56
Sep 26 06:28:53 dist-test-slave-fkbn krb5kdc[19162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404132, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:53.024137 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.014641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:55415 (local address 127.2.39.254:34647)
0926 06:28:53.014683 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.014684 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.014906 (+ 222us) server_negotiation.cc:407] Connection header received
0926 06:28:53.015446 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.015447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.015459 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.015524 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:53.016455 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.017030 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.018446 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.018757 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.018830 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.021116 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:53.021117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:53.021118 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:53.021141 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:53.022778 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:53.023193 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:53.023194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:53.023194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:53.023257 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:53.023512 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:53.023513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:53.023513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:53.023627 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:53.023700 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.023965 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.024047 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:28:53.024309 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.014611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:34647 (local address 127.2.39.194:55415)
0926 06:28:53.014781 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.014834 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.015333 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.015594 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.015598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.016167 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:53.016350 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.016354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.017127 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.017128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.018363 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.018365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.018837 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.018838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.018929 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.019430 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:53.019438 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:53.021038 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:53.022880 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:53.022887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:53.022888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:53.023108 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:53.023322 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:53.023322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:53.023323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:53.023432 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:53.023691 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:53.023697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:53.023831 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.023937 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.024083 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260926 06:28:53.024765 19532 heartbeater.cc:347] Connected to a master server at 127.2.39.254:34647
I20260926 06:28:53.024839 19532 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:53.025017 19532 heartbeater.cc:511] Master 127.2.39.254:34647 requested a full tablet report, sending...
I20260926 06:28:53.025408 19206 ts_manager.cc:194] Registered new tserver with Master: 6153f4cdcedb4641aa4a837924b46a75 (127.2.39.194:38029)
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
I20260926 06:28:53.025995 19206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:55415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:28:53.045068 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34647
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:53.124966 19537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:53.125152 19537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:53.125174 19537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:53.126260 19537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:53.126294 19537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:53.126319 19537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:28:53.127642 19537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:53.127808 19537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:53.127966 19537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:28:53.132957 19537 server_base.cc:1034] running on GCE node
W20260926 06:28:53.132928 19545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:53.132928 19542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:53.132920 19543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:53.133386 19537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:53.133688 19537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:53.134840 19537 hybrid_clock.cc:648] HybridClock initialized: now 1790404133134817 us; error 35 us; skew 500 ppm
Sep 26 06:28:53 dist-test-slave-fkbn krb5kdc[19162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404133, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:53.137110 19537 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:28:53.137539 19537 webserver.cc:460] Webserver started at http://127.2.39.195:34239/ using document root <none> and password file <none>
I20260926 06:28:53.137781 19537 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:53.137849 19537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:53.137990 19537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:53.138983 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "37d6a640a6c94857b62783d170b57c74"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "0ed0deb2243a8f36d426024f4b8cef8d"
server_key_iv: "2070f55cde6374f1ab3165955b435b0a"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.139341 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "37d6a640a6c94857b62783d170b57c74"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "0ed0deb2243a8f36d426024f4b8cef8d"
server_key_iv: "2070f55cde6374f1ab3165955b435b0a"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.140607 19537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:28:53.141340 19552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.141517 19537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:28:53.141624 19537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "37d6a640a6c94857b62783d170b57c74"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "0ed0deb2243a8f36d426024f4b8cef8d"
server_key_iv: "2070f55cde6374f1ab3165955b435b0a"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.141712 19537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:53.157015 19537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:53.158814 19537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:53.158969 19537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:53.159212 19537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:53.159531 19537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:53.159587 19537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.159626 19537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:53.159654 19537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.165114 19537 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:42213
I20260926 06:28:53.165216 19665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:42213 every 8 connection(s)
I20260926 06:28:53.165516 19537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:28:53.169628 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 19537
I20260926 06:28:53.169747 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:28:53.169920 2207 external_mini_cluster.cc:1428] Setting key 24faf4980e10a51cfe0c286561a6c5a7
Sep 26 06:28:53 dist-test-slave-fkbn krb5kdc[19162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404133, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:53.174698 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.166263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:39181 (local address 127.2.39.254:34647)
0926 06:28:53.166325 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.166327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.166478 (+ 151us) server_negotiation.cc:407] Connection header received
0926 06:28:53.166969 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.166970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.166993 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.167053 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:53.167933 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.168422 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.169354 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.169535 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.169664 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.171891 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:53.171893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:53.171893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:53.171901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:53.173312 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:53.173729 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:53.173730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:53.173730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:53.173807 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:53.174076 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:53.174077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:53.174077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:53.174198 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:53.174250 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.174569 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.174639 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 06:28:53.174966 19668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.166201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:34647 (local address 127.2.39.195:39181)
0926 06:28:53.166354 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.166395 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.166854 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.167130 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.167133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.167671 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:53.167851 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.167855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.168497 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.168499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.169281 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.169283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.169682 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.169683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.169850 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.170397 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:53.170405 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:53.171792 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:53.173380 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:53.173385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:53.173387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:53.173667 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:53.173913 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:53.173914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:53.173915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:53.174017 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:53.174254 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:53.174258 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:53.174364 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.174533 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.174701 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260926 06:28:53.175379 19666 heartbeater.cc:347] Connected to a master server at 127.2.39.254:34647
I20260926 06:28:53.175495 19666 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:53.175722 19666 heartbeater.cc:511] Master 127.2.39.254:34647 requested a full tablet report, sending...
I20260926 06:28:53.176081 19206 ts_manager.cc:194] Registered new tserver with Master: 37d6a640a6c94857b62783d170b57c74 (127.2.39.195:42213)
I20260926 06:28:53.176489 19206 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:39181
I20260926 06:28:53.181500 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:28:53.188884 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.182887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.2.39.254:34647)
0926 06:28:53.182971 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.182974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.183075 (+ 101us) server_negotiation.cc:407] Connection header received
0926 06:28:53.183153 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.183154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.183163 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.183219 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:53.183784 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.184293 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.185061 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.185257 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.185339 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.186591 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:53.186592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:53.186593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:53.186599 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:53.187740 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:53.188088 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:53.188089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:53.188089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:53.188146 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:53.188356 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:53.188357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:53.188357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:53.188505 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:53.188617 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.188712 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.188788 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 06:28:53.190439 19206 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:28:53.191998 19206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:28:53.198684 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.194209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:42213 (local address 127.0.0.1:32938)
0926 06:28:53.194588 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.194630 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.194703 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.194969 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.194971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.194973 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:53.195052 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.195054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.196338 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.196339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.197477 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.197480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.198365 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.198367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.198459 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.198474 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.198537 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.198589 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":36,"thread_start_us":71,"threads_started":1}
I20260926 06:28:53.198719 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.194465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.2.39.193:44995)
0926 06:28:53.194598 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.194602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.195116 (+ 514us) server_negotiation.cc:407] Connection header received
0926 06:28:53.195161 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.195165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.195181 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.195261 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:53.195380 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.196174 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.197739 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.198391 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.198470 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.198522 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.198604 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.198642 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260926 06:28:53.198868 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.194398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52812 (local address 127.2.39.194:38029)
0926 06:28:53.194627 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.194631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.194880 (+ 249us) server_negotiation.cc:407] Connection header received
0926 06:28:53.195398 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.195401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.195416 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.195491 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:53.195603 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.196622 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.197883 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.198399 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.198466 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.198530 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.198731 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.198776 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
I20260926 06:28:53.199019 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.194371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32938 (local address 127.2.39.195:42213)
0926 06:28:53.194552 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.194557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.194684 (+ 127us) server_negotiation.cc:407] Connection header received
0926 06:28:53.194796 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.194798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.194816 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.194886 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:53.195134 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.196229 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.197559 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.198268 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.198343 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.198398 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.198892 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.198934 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260926 06:28:53.198755 19675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.194221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:44995 (local address 127.0.0.1:37904)
0926 06:28:53.194980 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.194989 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.195049 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.195257 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.195257 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.195259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:53.195299 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.195301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.196283 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.196284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.197658 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.197661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.198473 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.198474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.198534 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.198538 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.198592 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.198631 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":69,"thread_start_us":56,"threads_started":1}
I20260926 06:28:53.198829 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.194209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:38029 (local address 127.0.0.1:52812)
0926 06:28:53.194802 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.194812 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.194863 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.195509 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.195509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.195511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:53.195543 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.195544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.196722 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.196723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.197809 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.197811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.198563 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.198563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.198613 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.198616 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.198661 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.198691 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"mutex_wait_us":2,"thread_start_us":66,"threads_started":1}
I20260926 06:28:53.199883 19600 tablet_service.cc:1467] Processing CreateTablet for tablet 98c581e88f514123b98becb63085914a (DEFAULT_TABLE table=test-table [id=edb3be92a007426f8d7fcd155fe6384d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:53.199949 19332 tablet_service.cc:1467] Processing CreateTablet for tablet 98c581e88f514123b98becb63085914a (DEFAULT_TABLE table=test-table [id=edb3be92a007426f8d7fcd155fe6384d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:53.200157 19600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c581e88f514123b98becb63085914a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:53.200266 19332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c581e88f514123b98becb63085914a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:53.200229 19466 tablet_service.cc:1467] Processing CreateTablet for tablet 98c581e88f514123b98becb63085914a (DEFAULT_TABLE table=test-table [id=edb3be92a007426f8d7fcd155fe6384d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:53.200501 19466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98c581e88f514123b98becb63085914a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:53.202708 19680 tablet_bootstrap.cc:492] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: Bootstrap starting.
I20260926 06:28:53.203323 19680 tablet_bootstrap.cc:654] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:53.203658 19682 tablet_bootstrap.cc:492] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: Bootstrap starting.
I20260926 06:28:53.203867 19680 log.cc:824] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:53.204288 19682 tablet_bootstrap.cc:654] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:53.204684 19680 tablet_bootstrap.cc:492] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: No bootstrap required, opened a new log
I20260926 06:28:53.204761 19680 ts_tablet_manager.cc:1397] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:28:53.204845 19682 log.cc:824] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:53.205168 19681 tablet_bootstrap.cc:492] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54: Bootstrap starting.
I20260926 06:28:53.205605 19682 tablet_bootstrap.cc:492] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: No bootstrap required, opened a new log
I20260926 06:28:53.205672 19682 ts_tablet_manager.cc:1397] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:28:53.205842 19681 tablet_bootstrap.cc:654] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:53.206424 19681 log.cc:824] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:53.206909 19680 raft_consensus.cc:348] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 } } peers { permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 } } peers { permanent_uuid: "569f1c4d975442bc89b83448535a3d54" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44995 } }
I20260926 06:28:53.207094 19680 raft_consensus.cc:374] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:53.207134 19680 raft_consensus.cc:729] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d6a640a6c94857b62783d170b57c74, State: Initialized, Role: FOLLOWER
I20260926 06:28:53.207156 19681 tablet_bootstrap.cc:492] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54: No bootstrap required, opened a new log
I20260926 06:28:53.207204 19681 ts_tablet_manager.cc:1397] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 06:28:53.207273 19680 consensus_queue.cc:260] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 } } peers { permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 } } peers { permanent_uuid: "569f1c4d975442bc89b83448535a3d54" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44995 } }
I20260926 06:28:53.207535 19680 ts_tablet_manager.cc:1428] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:28:53.207649 19666 heartbeater.cc:503] Master 127.2.39.254:34647 was elected leader, sending a full tablet report...
I20260926 06:28:53.207650 19682 raft_consensus.cc:348] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 } } peers { permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 } } peers { permanent_uuid: "569f1c4d975442bc89b83448535a3d54" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44995 } }
I20260926 06:28:53.207832 19682 raft_consensus.cc:374] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:53.207873 19682 raft_consensus.cc:729] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6153f4cdcedb4641aa4a837924b46a75, State: Initialized, Role: FOLLOWER
I20260926 06:28:53.207978 19682 consensus_queue.cc:260] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 } } peers { permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 } } peers { permanent_uuid: "569f1c4d975442bc89b83448535a3d54" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44995 } }
I20260926 06:28:53.208214 19682 ts_tablet_manager.cc:1428] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:28:53.208335 19532 heartbeater.cc:503] Master 127.2.39.254:34647 was elected leader, sending a full tablet report...
I20260926 06:28:53.209095 19681 raft_consensus.cc:348] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 } } peers { permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 } } peers { permanent_uuid: "569f1c4d975442bc89b83448535a3d54" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44995 } }
I20260926 06:28:53.209239 19681 raft_consensus.cc:374] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:53.209280 19681 raft_consensus.cc:729] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569f1c4d975442bc89b83448535a3d54, State: Initialized, Role: FOLLOWER
I20260926 06:28:53.209367 19681 consensus_queue.cc:260] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 } } peers { permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 } } peers { permanent_uuid: "569f1c4d975442bc89b83448535a3d54" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44995 } }
I20260926 06:28:53.209558 19681 ts_tablet_manager.cc:1428] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 06:28:53.209642 19398 heartbeater.cc:503] Master 127.2.39.254:34647 was elected leader, sending a full tablet report...
I20260926 06:28:53.223641 19688 raft_consensus.cc:482] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:53.223726 19688 raft_consensus.cc:504] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 } } peers { permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 } } peers { permanent_uuid: "569f1c4d975442bc89b83448535a3d54" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44995 } }
I20260926 06:28:53.224013 19688 leader_election.cc:288] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37d6a640a6c94857b62783d170b57c74 (127.2.39.195:42213), 6153f4cdcedb4641aa4a837924b46a75 (127.2.39.194:38029)
I20260926 06:28:53.226933 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.224134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:42213 (local address 127.2.39.193:40625)
0926 06:28:53.224235 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.224248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.224290 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.224410 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.224410 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.224412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:53.224481 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.224483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.225197 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.225198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.226093 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.226096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.226664 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.226665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.226745 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.226749 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.226812 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.226857 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":29}
I20260926 06:28:53.226972 19677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.224254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:40625 (local address 127.2.39.195:42213)
0926 06:28:53.224296 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.224299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.224316 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:28:53.224340 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.224341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.224350 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.224401 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:53.224564 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.225121 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.226175 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.226596 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.226650 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.226657 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.226846 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.226886 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:28:53.227141 19678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.224209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:37497 (local address 127.2.39.194:38029)
0926 06:28:53.224290 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.224293 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.224518 (+ 225us) server_negotiation.cc:407] Connection header received
0926 06:28:53.224544 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.224545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.224557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.224610 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:53.224748 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.225210 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.226236 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.226789 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.226852 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.226860 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.227042 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.227084 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:28:53.227350 19689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.224158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:38029 (local address 127.2.39.193:37497)
0926 06:28:53.224406 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.224416 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.224481 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.224618 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.224618 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.224620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:53.224685 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.224686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.225289 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.225290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.226148 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.226152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.226875 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.226876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.226944 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.226948 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.227233 (+ 285us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.227280 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":75,"thread_start_us":123,"threads_started":1}
I20260926 06:28:53.227674 19486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c581e88f514123b98becb63085914a" candidate_uuid: "569f1c4d975442bc89b83448535a3d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6153f4cdcedb4641aa4a837924b46a75" is_pre_election: true
I20260926 06:28:53.227707 19620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c581e88f514123b98becb63085914a" candidate_uuid: "569f1c4d975442bc89b83448535a3d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37d6a640a6c94857b62783d170b57c74" is_pre_election: true
I20260926 06:28:53.227836 19486 raft_consensus.cc:2454] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 569f1c4d975442bc89b83448535a3d54 in term 0.
I20260926 06:28:53.227847 19620 raft_consensus.cc:2454] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 569f1c4d975442bc89b83448535a3d54 in term 0.
I20260926 06:28:53.228066 19288 leader_election.cc:302] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 569f1c4d975442bc89b83448535a3d54, 6153f4cdcedb4641aa4a837924b46a75; no voters:
I20260926 06:28:53.228201 19688 raft_consensus.cc:2781] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:28:53.228255 19688 raft_consensus.cc:482] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:28:53.228300 19688 raft_consensus.cc:3037] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:53.228953 19688 raft_consensus.cc:504] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 } } peers { permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 } } peers { permanent_uuid: "569f1c4d975442bc89b83448535a3d54" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44995 } }
I20260926 06:28:53.229089 19688 leader_election.cc:288] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [CANDIDATE]: Term 1 election: Requested vote from peers 37d6a640a6c94857b62783d170b57c74 (127.2.39.195:42213), 6153f4cdcedb4641aa4a837924b46a75 (127.2.39.194:38029)
I20260926 06:28:53.229235 19486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c581e88f514123b98becb63085914a" candidate_uuid: "569f1c4d975442bc89b83448535a3d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6153f4cdcedb4641aa4a837924b46a75"
I20260926 06:28:53.229249 19620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98c581e88f514123b98becb63085914a" candidate_uuid: "569f1c4d975442bc89b83448535a3d54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37d6a640a6c94857b62783d170b57c74"
I20260926 06:28:53.229331 19486 raft_consensus.cc:3037] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:53.229327 19620 raft_consensus.cc:3037] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:53.230186 19620 raft_consensus.cc:2454] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 569f1c4d975442bc89b83448535a3d54 in term 1.
I20260926 06:28:53.230249 19486 raft_consensus.cc:2454] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 569f1c4d975442bc89b83448535a3d54 in term 1.
I20260926 06:28:53.230384 19286 leader_election.cc:302] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37d6a640a6c94857b62783d170b57c74, 569f1c4d975442bc89b83448535a3d54; no voters:
I20260926 06:28:53.230504 19688 raft_consensus.cc:2781] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:53.230696 19688 raft_consensus.cc:686] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [term 1 LEADER]: Becoming Leader. State: Replica: 569f1c4d975442bc89b83448535a3d54, State: Running, Role: LEADER
I20260926 06:28:53.230796 19688 consensus_queue.cc:237] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 } } peers { permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 } } peers { permanent_uuid: "569f1c4d975442bc89b83448535a3d54" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44995 } }
I20260926 06:28:53.231621 19207 catalog_manager.cc:5495] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 569f1c4d975442bc89b83448535a3d54 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "569f1c4d975442bc89b83448535a3d54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "569f1c4d975442bc89b83448535a3d54" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 44995 } health_report { overall_health: HEALTHY } } }
I20260926 06:28:53.243449 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.241291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.2.39.193:44995)
0926 06:28:53.241344 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.241346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.241363 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:28:53.241388 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.241389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.241403 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.241472 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:28:53.241641 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.242101 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.242826 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.242997 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.243042 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.243169 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:28:53.243254 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:28:53.243310 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.243339 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.243353 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:28:53.245112 19486 raft_consensus.cc:1261] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75 [term 1 FOLLOWER]: Refusing update from remote peer 569f1c4d975442bc89b83448535a3d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:53.245136 19620 raft_consensus.cc:1261] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74 [term 1 FOLLOWER]: Refusing update from remote peer 569f1c4d975442bc89b83448535a3d54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:53.245469 19690 consensus_queue.cc:1035] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6153f4cdcedb4641aa4a837924b46a75" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:53.245613 19688 consensus_queue.cc:1035] T 98c581e88f514123b98becb63085914a P 569f1c4d975442bc89b83448535a3d54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d6a640a6c94857b62783d170b57c74" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:53.247792 19693 mvcc.cc:205] Tried to move back new op lower bound from 7333495329769549824 to 7333495329714118656. Current Snapshot: MvccSnapshot[applied={T|T < 7333495329769549824}]
I20260926 06:28:53.248450 19694 mvcc.cc:205] Tried to move back new op lower bound from 7333495329769549824 to 7333495329714118656. Current Snapshot: MvccSnapshot[applied={T|T < 7333495329769549824}]
I20260926 06:28:53.248719 19695 mvcc.cc:205] Tried to move back new op lower bound from 7333495329769549824 to 7333495329714118656. Current Snapshot: MvccSnapshot[applied={T|T < 7333495329769549824}]
I20260926 06:28:53.250108 19206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:53.250204 19206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:53.250914 19206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a5d86b76a8f4bb9835ecef9c4bf7946: Sending DeleteTablet for 3 replicas of tablet 98c581e88f514123b98becb63085914a
I20260926 06:28:53.251255 19600 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c581e88f514123b98becb63085914a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:53 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:32938
I20260926 06:28:53.251257 19332 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c581e88f514123b98becb63085914a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:53 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:37904
I20260926 06:28:53.251300 19466 tablet_service.cc:1514] Processing DeleteTablet for tablet 98c581e88f514123b98becb63085914a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:53 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:52812
I20260926 06:28:53.251490 19704 tablet_replica.cc:330] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: stopping tablet replica
I20260926 06:28:53.251577 19704 raft_consensus.cc:2229] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:53.251744 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 19269
I20260926 06:28:53.251812 19704 raft_consensus.cc:2258] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:53.252311 19702 tablet_replica.cc:330] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: stopping tablet replica
I20260926 06:28:53.252396 19702 raft_consensus.cc:2229] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:53.252455 19702 raft_consensus.cc:2258] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:53.253089 19702 ts_tablet_manager.cc:1905] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:53.254370 19704 ts_tablet_manager.cc:1905] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:53.254989 19702 ts_tablet_manager.cc:1918] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:53.255087 19702 log.cc:1196] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/98c581e88f514123b98becb63085914a
I20260926 06:28:53.255326 19702 ts_tablet_manager.cc:1939] T 98c581e88f514123b98becb63085914a P 37d6a640a6c94857b62783d170b57c74: Deleting consensus metadata
I20260926 06:28:53.256224 19704 ts_tablet_manager.cc:1918] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:53.256296 19704 log.cc:1196] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/98c581e88f514123b98becb63085914a
I20260926 06:28:53.256552 19704 ts_tablet_manager.cc:1939] T 98c581e88f514123b98becb63085914a P 6153f4cdcedb4641aa4a837924b46a75: Deleting consensus metadata
I20260926 06:28:53.256646 19192 catalog_manager.cc:4841] TS 37d6a640a6c94857b62783d170b57c74 (127.2.39.195:42213): tablet 98c581e88f514123b98becb63085914a (table test-table [id=edb3be92a007426f8d7fcd155fe6384d]) successfully deleted
I20260926 06:28:53.256968 19194 catalog_manager.cc:4841] TS 6153f4cdcedb4641aa4a837924b46a75 (127.2.39.194:38029): tablet 98c581e88f514123b98becb63085914a (table test-table [id=edb3be92a007426f8d7fcd155fe6384d]) successfully deleted
W20260926 06:28:53.258222 19191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:44995 (error 108)
I20260926 06:28:53.258450 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 19403
I20260926 06:28:53.259029 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.258562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:44995 (local address 127.0.0.1:37934)
0926 06:28:53.258958 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.259005 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:44995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378}
W20260926 06:28:53.259171 19191 catalog_manager.cc:4584] TS 569f1c4d975442bc89b83448535a3d54 (127.2.39.193:44995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98c581e88f514123b98becb63085914a: Network error: Client connection negotiation failed: client connection to 127.2.39.193:44995: connect: Connection refused (error 111)
I20260926 06:28:53.263940 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 19537
I20260926 06:28:53.268339 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 19175
2026-09-26T06:28:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1886 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T06:28:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:28:53Z Disabled control of system clock
I20260926 06:28:53.295410 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:41087
--webserver_interface=127.2.39.254
--webserver_port=0
--builtin_ntp_servers=127.2.39.212:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:41087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:28:53.374704 19709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:53.374903 19709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:53.374923 19709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:53.375995 19709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:28:53.376029 19709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:53.376042 19709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:53.376055 19709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:28:53.376066 19709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:28:53.377286 19709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:41087
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:41087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:53.377460 19709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:53.377704 19709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:53.382371 19717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:53.382488 19715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:53.382367 19714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:53.382570 19709 server_base.cc:1034] running on GCE node
I20260926 06:28:53.382797 19709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:53.383028 19709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:53.384181 19709 hybrid_clock.cc:648] HybridClock initialized: now 1790404133384151 us; error 27 us; skew 500 ppm
I20260926 06:28:53.385296 19709 webserver.cc:460] Webserver started at http://127.2.39.254:39135/ using document root <none> and password file <none>
I20260926 06:28:53.385531 19709 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:53.385623 19709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:53.385789 19709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:53.386834 19709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "c40b36d2bd41428fb8ff632f78cc6512"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "9b3e7f82c9cd9338ecb94ade9f5b9640"
server_key_iv: "2ae8d3c8dcbb975fde26794ccb5502b0"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.387179 19709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "c40b36d2bd41428fb8ff632f78cc6512"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "9b3e7f82c9cd9338ecb94ade9f5b9640"
server_key_iv: "2ae8d3c8dcbb975fde26794ccb5502b0"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.388396 19709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:28:53.389081 19723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.389248 19709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:28:53.389345 19709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "c40b36d2bd41428fb8ff632f78cc6512"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "9b3e7f82c9cd9338ecb94ade9f5b9640"
server_key_iv: "2ae8d3c8dcbb975fde26794ccb5502b0"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.389421 19709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:53.402457 19709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:53.402755 19709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:53.402902 19709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:53.406740 19709 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:41087
I20260926 06:28:53.406773 19775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:41087 every 8 connection(s)
I20260926 06:28:53.407125 19709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:28:53.408006 19776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:53.409698 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 19709
I20260926 06:28:53.409802 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:28:53.409981 2207 external_mini_cluster.cc:1428] Setting key b11455a8e3e7b912c69360f4b571bc6a
I20260926 06:28:53.410481 19776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512: Bootstrap starting.
I20260926 06:28:53.411072 19776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:53.411397 19776 log.cc:824] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:53.412216 19776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512: No bootstrap required, opened a new log
I20260926 06:28:53.414407 19776 raft_consensus.cc:348] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40b36d2bd41428fb8ff632f78cc6512" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41087 } }
I20260926 06:28:53.414549 19776 raft_consensus.cc:374] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:53.414573 19776 raft_consensus.cc:729] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40b36d2bd41428fb8ff632f78cc6512, State: Initialized, Role: FOLLOWER
I20260926 06:28:53.414721 19776 consensus_queue.cc:260] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40b36d2bd41428fb8ff632f78cc6512" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41087 } }
I20260926 06:28:53.414817 19776 raft_consensus.cc:388] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:28:53.414856 19776 raft_consensus.cc:482] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:28:53.414903 19776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:53.415606 19776 raft_consensus.cc:504] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40b36d2bd41428fb8ff632f78cc6512" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41087 } }
I20260926 06:28:53.415726 19776 leader_election.cc:302] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c40b36d2bd41428fb8ff632f78cc6512; no voters:
I20260926 06:28:53.415905 19776 leader_election.cc:288] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:28:53.415934 19781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:53.416152 19781 raft_consensus.cc:686] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [term 1 LEADER]: Becoming Leader. State: Replica: c40b36d2bd41428fb8ff632f78cc6512, State: Running, Role: LEADER
I20260926 06:28:53.416234 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.410492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.2.39.254:41087)
0926 06:28:53.410706 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.410713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.410737 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:28:53.411194 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.411203 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.411274 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.411390 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:53.411779 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.412822 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.413992 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.414544 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.414726 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.415526 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:53.415529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:53.415530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:53.415686 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:53.415741 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:53.415756 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:53.415810 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.415915 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.415996 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260926 06:28:53.416261 19776 sys_catalog.cc:564] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:28:53.416493 19781 consensus_queue.cc:237] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40b36d2bd41428fb8ff632f78cc6512" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41087 } }
I20260926 06:28:53.416970 19782 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40b36d2bd41428fb8ff632f78cc6512. Latest consensus state: current_term: 1 leader_uuid: "c40b36d2bd41428fb8ff632f78cc6512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40b36d2bd41428fb8ff632f78cc6512" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41087 } } }
I20260926 06:28:53.417068 19782 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:53.416985 19783 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c40b36d2bd41428fb8ff632f78cc6512" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c40b36d2bd41428fb8ff632f78cc6512" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41087 } } }
I20260926 06:28:53.417111 19783 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:53.417253 19790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:28:53.417757 19790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:28:53.419277 19790 catalog_manager.cc:1347] Generated new cluster ID: e6fe7ca4d13b45b18e6584a308062462
I20260926 06:28:53.419351 19790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:28:53.439280 19790 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:28:53.439884 19790 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:28:53.443120 19790 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512: Generated new TSK 0
I20260926 06:28:53.443320 19790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:28:53.447064 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41087
--builtin_ntp_servers=127.2.39.212:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:28:53.528733 19800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:53.528923 19800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:53.528957 19800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:53.530149 19800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:53.530189 19800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:53.530215 19800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:28:53.531610 19800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:53.531847 19800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:53.532146 19800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:53.537774 19808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:53.537849 19806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:53.537786 19805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:53.538060 19800 server_base.cc:1034] running on GCE node
I20260926 06:28:53.538291 19800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:53.538548 19800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:53.539697 19800 hybrid_clock.cc:648] HybridClock initialized: now 1790404133539677 us; error 37 us; skew 500 ppm
I20260926 06:28:53.541028 19800 webserver.cc:460] Webserver started at http://127.2.39.193:44687/ using document root <none> and password file <none>
I20260926 06:28:53.541280 19800 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:53.541325 19800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:53.541491 19800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:53.542550 19800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "54058034cc3742f7889db661e33e389e"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "dbb196643606bfbc80cb2192684aa451"
server_key_iv: "7780cd7e87fc2048675dfb32ea61362f"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.542891 19800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "54058034cc3742f7889db661e33e389e"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "dbb196643606bfbc80cb2192684aa451"
server_key_iv: "7780cd7e87fc2048675dfb32ea61362f"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.544154 19800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:28:53.544855 19814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.545054 19800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:28:53.545141 19800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "54058034cc3742f7889db661e33e389e"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "dbb196643606bfbc80cb2192684aa451"
server_key_iv: "7780cd7e87fc2048675dfb32ea61362f"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.545209 19800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:53.564604 19800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:53.564929 19800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:53.565091 19800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:53.565323 19800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:53.565737 19800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:53.565800 19800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.565843 19800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:53.565871 19800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.571666 19800 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:42575
I20260926 06:28:53.571728 19927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:42575 every 8 connection(s)
I20260926 06:28:53.572104 19800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:28:53.577127 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.572875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:59883 (local address 127.2.39.254:41087)
0926 06:28:53.572938 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.572940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.573197 (+ 257us) server_negotiation.cc:407] Connection header received
0926 06:28:53.573758 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.573759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.573768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.573824 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:53.574471 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.574962 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.575922 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.576100 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.576151 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.576896 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:53.576898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:53.576898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:53.576902 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:53.576916 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:53.576920 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:53.576965 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.577050 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.577071 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:28:53.577358 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.572851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41087 (local address 127.2.39.193:59883)
0926 06:28:53.573065 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.573109 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.573660 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.573889 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.573892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.573901 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:53.574396 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.574400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.575054 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.575055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.575845 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.575847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.576174 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.576175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.576281 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.576766 (+ 485us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:28:53.576766 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:53.576795 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:28:53.576816 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:53.576974 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:53.576979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:53.576982 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.577037 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.577113 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":120,"threads_started":1}
I20260926 06:28:53.577984 19928 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41087
I20260926 06:28:53.578063 19928 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:53.578207 19928 heartbeater.cc:511] Master 127.2.39.254:41087 requested a full tablet report, sending...
I20260926 06:28:53.578687 19740 ts_manager.cc:194] Registered new tserver with Master: 54058034cc3742f7889db661e33e389e (127.2.39.193:42575)
I20260926 06:28:53.579299 19740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.193:59883
I20260926 06:28:53.581773 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 19800
I20260926 06:28:53.581853 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:28:53.581995 2207 external_mini_cluster.cc:1428] Setting key f19bbc4e1c2c9596aae10bb842608e7b
I20260926 06:28:53.582815 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41087
--builtin_ntp_servers=127.2.39.212:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:28:53.666718 19931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:53.666944 19931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:53.666980 19931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:53.668169 19931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:53.668236 19931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:53.668278 19931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:28:53.669734 19931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:53.669989 19931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:53.670279 19931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:53.675232 19939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:53.675240 19937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:53.675271 19931 server_base.cc:1034] running on GCE node
W20260926 06:28:53.675240 19936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:53.675621 19931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:53.675859 19931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:53.677032 19931 hybrid_clock.cc:648] HybridClock initialized: now 1790404133677021 us; error 27 us; skew 500 ppm
I20260926 06:28:53.684309 19931 webserver.cc:460] Webserver started at http://127.2.39.194:43925/ using document root <none> and password file <none>
I20260926 06:28:53.684571 19931 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:53.684644 19931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:53.684793 19931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:53.685986 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "651ed538c97641e6896952b6bf660b7c"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "08756afe86f73cefeee938e5687c1377"
server_key_iv: "6d2659a5054f4a5391e4d0ffe3c99d16"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.686336 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "651ed538c97641e6896952b6bf660b7c"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "08756afe86f73cefeee938e5687c1377"
server_key_iv: "6d2659a5054f4a5391e4d0ffe3c99d16"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.687683 19931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:28:53.688406 19945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.688576 19931 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:28:53.688679 19931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "651ed538c97641e6896952b6bf660b7c"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "08756afe86f73cefeee938e5687c1377"
server_key_iv: "6d2659a5054f4a5391e4d0ffe3c99d16"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.688750 19931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:53.698899 19931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:53.699198 19931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:53.699345 19931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:53.699572 19931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:53.699884 19931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:53.699939 19931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.699985 19931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:53.700030 19931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.705634 19931 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:41833
I20260926 06:28:53.705684 20058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:41833 every 8 connection(s)
I20260926 06:28:53.705991 19931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:28:53.707767 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 19931
I20260926 06:28:53.707872 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:28:53.708122 2207 external_mini_cluster.cc:1428] Setting key 225f40d4acdd16c5c4c312cf4256395d
I20260926 06:28:53.709132 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41087
--builtin_ntp_servers=127.2.39.212:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 06:28:53.711972 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.706651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:36713 (local address 127.2.39.254:41087)
0926 06:28:53.706716 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.706718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.706954 (+ 236us) server_negotiation.cc:407] Connection header received
0926 06:28:53.707398 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.707399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.707407 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.707462 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:53.708349 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.708928 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.710442 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.710779 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.710856 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.711731 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:53.711732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:53.711732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:53.711736 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:53.711751 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:53.711755 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:53.711800 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.711893 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.711915 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:28:53.712244 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.706653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41087 (local address 127.2.39.194:36713)
0926 06:28:53.706828 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.706866 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.707321 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.707521 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.707524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.707532 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:53.708248 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.708255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.709013 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.709014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.710342 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.710346 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.710864 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.710865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.710977 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.711577 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:28:53.711577 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:53.711613 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:28:53.711640 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:53.711814 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:53.711817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:53.711820 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.711884 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.711996 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260926 06:28:53.712738 20059 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41087
I20260926 06:28:53.712837 20059 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:53.713055 20059 heartbeater.cc:511] Master 127.2.39.254:41087 requested a full tablet report, sending...
I20260926 06:28:53.713434 19740 ts_manager.cc:194] Registered new tserver with Master: 651ed538c97641e6896952b6bf660b7c (127.2.39.194:41833)
I20260926 06:28:53.713850 19740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.194:36713
W20260926 06:28:53.790251 20062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:53.790433 20062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:53.790462 20062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:53.791682 20062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:53.791728 20062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:53.791779 20062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:28:53.793220 20062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:53.793395 20062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:53.793650 20062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:28:53.798496 20062 server_base.cc:1034] running on GCE node
W20260926 06:28:53.798516 20067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:53.798522 20068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:53.798544 20070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:53.798967 20062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:53.799154 20062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:53.800312 20062 hybrid_clock.cc:648] HybridClock initialized: now 1790404133800297 us; error 29 us; skew 500 ppm
I20260926 06:28:53.801476 20062 webserver.cc:460] Webserver started at http://127.2.39.195:44325/ using document root <none> and password file <none>
I20260926 06:28:53.801749 20062 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:53.801818 20062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:53.801982 20062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:53.802976 20062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "96294b0dea84415792c72ba6fbe43236"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "0566fb5e8c7cd5be0d9ef4372509eaeb"
server_key_iv: "9d8188a9d631c617ab4bdbbd3b37ab87"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.803316 20062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "96294b0dea84415792c72ba6fbe43236"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "0566fb5e8c7cd5be0d9ef4372509eaeb"
server_key_iv: "9d8188a9d631c617ab4bdbbd3b37ab87"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.804572 20062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:28:53.805266 20076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.805435 20062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:28:53.805528 20062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "96294b0dea84415792c72ba6fbe43236"
format_stamp: "Formatted at 2026-09-26 06:28:53 on dist-test-slave-fkbn"
server_key: "0566fb5e8c7cd5be0d9ef4372509eaeb"
server_key_iv: "9d8188a9d631c617ab4bdbbd3b37ab87"
server_key_version: "encryptionkey@0"
I20260926 06:28:53.805629 20062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:53.818631 20062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:53.819028 20062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:53.819207 20062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:53.819465 20062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:53.819849 20062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:53.819918 20062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.819967 20062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:53.820003 20062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:53.826891 20062 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:42819
I20260926 06:28:53.826926 20189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:42819 every 8 connection(s)
I20260926 06:28:53.827270 20062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:28:53.832713 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.828031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:60929 (local address 127.2.39.254:41087)
0926 06:28:53.828107 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.828108 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.828367 (+ 259us) server_negotiation.cc:407] Connection header received
0926 06:28:53.828923 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.828924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.828933 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.828985 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:53.829755 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.830235 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.831397 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.831583 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.831632 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.832476 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:53.832477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:53.832478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:53.832482 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:53.832495 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:53.832499 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:53.832540 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.832635 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.832656 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:28:53.832983 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.828026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41087 (local address 127.2.39.195:60929)
0926 06:28:53.828235 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.828286 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.828800 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.829033 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.829037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.829045 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:53.829642 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.829648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.830409 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.830410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.831311 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.831314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.831662 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.831663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.831775 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.832334 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:28:53.832335 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:53.832364 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:28:53.832387 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:53.832558 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:53.832562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:53.832564 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.832625 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.832705 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260926 06:28:53.833547 20190 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41087
I20260926 06:28:53.833690 20190 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:53.833868 20190 heartbeater.cc:511] Master 127.2.39.254:41087 requested a full tablet report, sending...
I20260926 06:28:53.834199 19740 ts_manager.cc:194] Registered new tserver with Master: 96294b0dea84415792c72ba6fbe43236 (127.2.39.195:42819)
I20260926 06:28:53.834583 19740 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.195:60929
I20260926 06:28:53.834574 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 20062
I20260926 06:28:53.834709 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:28:53.834962 2207 external_mini_cluster.cc:1428] Setting key 2f4cd174a656ff9427b4de1d0f23c0c1
I20260926 06:28:53.835829 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:28:53.839704 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.837199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.2.39.254:41087)
0926 06:28:53.837255 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.837257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.837349 (+ 92us) server_negotiation.cc:407] Connection header received
0926 06:28:53.837395 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.837396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.837403 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.837459 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:53.837635 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.838075 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.838761 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.838926 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.838981 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.839532 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:53.839533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:53.839533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:53.839537 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:53.839548 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:53.839552 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:53.839595 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.839642 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.839658 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 06:28:53.841236 19740 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:28:53.842801 19740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:28:53.848737 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.844809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:42819 (local address 127.0.0.1:34616)
0926 06:28:53.844966 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.845001 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.845060 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.845426 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.845428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.845430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:53.845498 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.845498 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.846520 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.846520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.847577 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.847580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.848462 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.848463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.848534 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.848546 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.848619 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.848669 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260926 06:28:53.848745 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.844883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.2.39.195:42819)
0926 06:28:53.845190 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.845193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.845202 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:28:53.845240 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.845242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.845254 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.845352 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:53.845562 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.846425 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.847669 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.848376 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.848462 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.848544 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.848632 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.848667 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":80,"threads_started":1}
I20260926 06:28:53.848968 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.844809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:42575 (local address 127.0.0.1:52736)
0926 06:28:53.845224 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.845234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.845292 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.845461 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.845462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.845463 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:53.845496 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.845498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.846895 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.846896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.847887 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.847889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.848760 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.848761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.848823 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.848827 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.848858 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.848892 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":7,"thread_start_us":56,"threads_started":1}
I20260926 06:28:53.849321 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.844980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.2.39.193:42575)
0926 06:28:53.845156 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.845161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.845305 (+ 144us) server_negotiation.cc:407] Connection header received
0926 06:28:53.845348 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.845350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.845364 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.845592 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:53.845668 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.846799 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.847961 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.848660 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.849093 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.849176 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.849200 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.849245 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260926 06:28:53.849335 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.845019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:41833 (local address 127.0.0.1:54100)
0926 06:28:53.845359 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:53.845366 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:53.845404 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:53.845864 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:53.845865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:53.845867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:53.845911 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.845912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.847073 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.847074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.848121 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:53.848123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.849125 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.849126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:53.849187 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.849191 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:53.849239 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:28:53.849272 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260926 06:28:53.849637 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:53.845205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.2.39.194:41833)
0926 06:28:53.845712 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:53.845718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:53.845728 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:28:53.845763 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:53.845765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:53.845780 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:53.845856 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:53.845986 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.846993 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.848190 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:53.848942 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:53.849003 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:53.849089 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:53.849491 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 06:28:53.849530 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":77,"threads_started":1}
I20260926 06:28:53.849670 20124 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4baa2ad3214b0c8fbc653fd415b24a (DEFAULT_TABLE table=test-table [id=f8b9ea5047b94de3b11b49664751e2c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:53.849888 19862 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4baa2ad3214b0c8fbc653fd415b24a (DEFAULT_TABLE table=test-table [id=f8b9ea5047b94de3b11b49664751e2c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:53.850065 20124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4baa2ad3214b0c8fbc653fd415b24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:53.850236 19862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4baa2ad3214b0c8fbc653fd415b24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:53.850292 19993 tablet_service.cc:1467] Processing CreateTablet for tablet 0a4baa2ad3214b0c8fbc653fd415b24a (DEFAULT_TABLE table=test-table [id=f8b9ea5047b94de3b11b49664751e2c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:53.850605 19993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a4baa2ad3214b0c8fbc653fd415b24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:53.852675 20204 tablet_bootstrap.cc:492] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: Bootstrap starting.
I20260926 06:28:53.853396 20204 tablet_bootstrap.cc:654] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:53.853482 20205 tablet_bootstrap.cc:492] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e: Bootstrap starting.
I20260926 06:28:53.853749 20206 tablet_bootstrap.cc:492] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: Bootstrap starting.
I20260926 06:28:53.853936 20204 log.cc:824] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:53.854147 20205 tablet_bootstrap.cc:654] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:53.854382 20206 tablet_bootstrap.cc:654] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:53.854471 20205 log.cc:824] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:53.854718 20206 log.cc:824] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:53.854763 20204 tablet_bootstrap.cc:492] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: No bootstrap required, opened a new log
I20260926 06:28:53.854839 20204 ts_tablet_manager.cc:1397] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:28:53.855206 20205 tablet_bootstrap.cc:492] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e: No bootstrap required, opened a new log
I20260926 06:28:53.855257 20205 ts_tablet_manager.cc:1397] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:28:53.855449 20206 tablet_bootstrap.cc:492] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: No bootstrap required, opened a new log
I20260926 06:28:53.855516 20206 ts_tablet_manager.cc:1397] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:28:53.856896 20205 raft_consensus.cc:348] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 } } peers { permanent_uuid: "96294b0dea84415792c72ba6fbe43236" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42819 } } peers { permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 } }
I20260926 06:28:53.857010 20205 raft_consensus.cc:374] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:53.857031 20205 raft_consensus.cc:729] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54058034cc3742f7889db661e33e389e, State: Initialized, Role: FOLLOWER
I20260926 06:28:53.857144 20205 consensus_queue.cc:260] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 } } peers { permanent_uuid: "96294b0dea84415792c72ba6fbe43236" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42819 } } peers { permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 } }
I20260926 06:28:53.857400 20205 ts_tablet_manager.cc:1428] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:28:53.857383 20204 raft_consensus.cc:348] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 } } peers { permanent_uuid: "96294b0dea84415792c72ba6fbe43236" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42819 } } peers { permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 } }
I20260926 06:28:53.857529 19928 heartbeater.cc:503] Master 127.2.39.254:41087 was elected leader, sending a full tablet report...
I20260926 06:28:53.857545 20204 raft_consensus.cc:374] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:53.857604 20204 raft_consensus.cc:729] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96294b0dea84415792c72ba6fbe43236, State: Initialized, Role: FOLLOWER
I20260926 06:28:53.857730 20204 consensus_queue.cc:260] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 } } peers { permanent_uuid: "96294b0dea84415792c72ba6fbe43236" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42819 } } peers { permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 } }
I20260926 06:28:53.857972 20204 ts_tablet_manager.cc:1428] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:28:53.858124 20190 heartbeater.cc:503] Master 127.2.39.254:41087 was elected leader, sending a full tablet report...
I20260926 06:28:53.858062 20206 raft_consensus.cc:348] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 } } peers { permanent_uuid: "96294b0dea84415792c72ba6fbe43236" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42819 } } peers { permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 } }
I20260926 06:28:53.858188 20206 raft_consensus.cc:374] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:53.858240 20206 raft_consensus.cc:729] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651ed538c97641e6896952b6bf660b7c, State: Initialized, Role: FOLLOWER
I20260926 06:28:53.858351 20206 consensus_queue.cc:260] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 } } peers { permanent_uuid: "96294b0dea84415792c72ba6fbe43236" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42819 } } peers { permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 } }
I20260926 06:28:53.858608 20206 ts_tablet_manager.cc:1428] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:28:53.858733 20059 heartbeater.cc:503] Master 127.2.39.254:41087 was elected leader, sending a full tablet report...
W20260926 06:28:53.956593 20060 tablet.cc:2369] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:28:54.003428 20211 raft_consensus.cc:482] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:54.003574 20211 raft_consensus.cc:504] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 } } peers { permanent_uuid: "96294b0dea84415792c72ba6fbe43236" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42819 } } peers { permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 } }
I20260926 06:28:54.003901 20211 leader_election.cc:288] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54058034cc3742f7889db661e33e389e (127.2.39.193:42575), 651ed538c97641e6896952b6bf660b7c (127.2.39.194:41833)
I20260926 06:28:54.006848 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:54.004253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:37703 (local address 127.2.39.193:42575)
0926 06:28:54.004320 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:54.004322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:54.004334 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:28:54.004366 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:54.004367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:54.004380 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:54.004491 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:54.004646 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:54.005133 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:54.006142 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:54.006503 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:54.006588 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:54.006596 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:54.006742 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 06:28:54.006777 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 06:28:54.006848 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:54.004146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:42575 (local address 127.2.39.195:37703)
0926 06:28:54.004206 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:54.004220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:54.004268 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:54.004484 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:54.004486 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:54.004489 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:54.004572 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:54.004573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:54.005222 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:54.005223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:54.006046 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:54.006049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:54.006586 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:54.006587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:54.006657 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:54.006659 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:28:54.006734 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 06:28:54.006775 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":1}
I20260926 06:28:54.007056 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:54.004358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:60909 (local address 127.2.39.194:41833)
0926 06:28:54.004428 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:54.004430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:54.004546 (+ 116us) server_negotiation.cc:407] Connection header received
0926 06:28:54.004613 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:54.004614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:54.004622 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:54.004694 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:54.004841 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:54.005292 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:54.006303 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:54.006663 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:54.006746 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:54.006753 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:54.006922 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 06:28:54.006984 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:28:54.007259 20213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:54.004272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:41833 (local address 127.2.39.195:60909)
0926 06:28:54.004475 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:54.004483 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:54.004553 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:54.004695 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:54.004695 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:54.004697 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:54.004766 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:54.004767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:54.005377 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:54.005378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:54.006208 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:54.006210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:54.006746 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:54.006747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:54.006828 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:54.006831 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:54.007141 (+ 310us) client_negotiation.cc:240] Negotiation successful
0926 06:28:54.007179 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":125,"threads_started":1}
I20260926 06:28:54.007244 19882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4baa2ad3214b0c8fbc653fd415b24a" candidate_uuid: "96294b0dea84415792c72ba6fbe43236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54058034cc3742f7889db661e33e389e" is_pre_election: true
I20260926 06:28:54.007411 19882 raft_consensus.cc:2454] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96294b0dea84415792c72ba6fbe43236 in term 0.
I20260926 06:28:54.007634 20078 leader_election.cc:302] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54058034cc3742f7889db661e33e389e, 96294b0dea84415792c72ba6fbe43236; no voters:
I20260926 06:28:54.007793 20211 raft_consensus.cc:2781] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:28:54.007862 20211 raft_consensus.cc:482] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:28:54.007880 20211 raft_consensus.cc:3037] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:54.007990 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4baa2ad3214b0c8fbc653fd415b24a" candidate_uuid: "96294b0dea84415792c72ba6fbe43236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "651ed538c97641e6896952b6bf660b7c" is_pre_election: true
I20260926 06:28:54.008154 20013 raft_consensus.cc:2454] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96294b0dea84415792c72ba6fbe43236 in term 0.
I20260926 06:28:54.008759 20211 raft_consensus.cc:504] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 } } peers { permanent_uuid: "96294b0dea84415792c72ba6fbe43236" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42819 } } peers { permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 } }
I20260926 06:28:54.008888 20211 leader_election.cc:288] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [CANDIDATE]: Term 1 election: Requested vote from peers 54058034cc3742f7889db661e33e389e (127.2.39.193:42575), 651ed538c97641e6896952b6bf660b7c (127.2.39.194:41833)
I20260926 06:28:54.009063 19882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4baa2ad3214b0c8fbc653fd415b24a" candidate_uuid: "96294b0dea84415792c72ba6fbe43236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54058034cc3742f7889db661e33e389e"
I20260926 06:28:54.009086 20013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a4baa2ad3214b0c8fbc653fd415b24a" candidate_uuid: "96294b0dea84415792c72ba6fbe43236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "651ed538c97641e6896952b6bf660b7c"
I20260926 06:28:54.009153 20013 raft_consensus.cc:3037] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:54.009171 19882 raft_consensus.cc:3037] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:54.009941 19882 raft_consensus.cc:2454] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96294b0dea84415792c72ba6fbe43236 in term 1.
I20260926 06:28:54.010102 20078 leader_election.cc:302] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54058034cc3742f7889db661e33e389e, 96294b0dea84415792c72ba6fbe43236; no voters:
I20260926 06:28:54.010195 20211 raft_consensus.cc:2781] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:54.010345 20211 raft_consensus.cc:686] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 1 LEADER]: Becoming Leader. State: Replica: 96294b0dea84415792c72ba6fbe43236, State: Running, Role: LEADER
I20260926 06:28:54.010435 20211 consensus_queue.cc:237] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 } } peers { permanent_uuid: "96294b0dea84415792c72ba6fbe43236" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42819 } } peers { permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 } }
I20260926 06:28:54.010452 20013 raft_consensus.cc:2454] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96294b0dea84415792c72ba6fbe43236 in term 1.
I20260926 06:28:54.011227 19739 catalog_manager.cc:5495] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96294b0dea84415792c72ba6fbe43236 (127.2.39.195). New cstate: current_term: 1 leader_uuid: "96294b0dea84415792c72ba6fbe43236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96294b0dea84415792c72ba6fbe43236" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 42819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 } health_report { overall_health: UNKNOWN } } }
I20260926 06:28:54.030018 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:54.027837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.2.39.195:42819)
0926 06:28:54.027887 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:54.027889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:54.027897 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:28:54.027957 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:54.027958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:54.027985 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:54.028044 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:28:54.028137 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:54.028598 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:54.029286 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:54.029463 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:54.029515 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:54.029683 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:28:54.029778 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:28:54.029838 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:54.029870 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 06:28:54.029884 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:28:54.031886 20013 raft_consensus.cc:1261] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c [term 1 FOLLOWER]: Refusing update from remote peer 96294b0dea84415792c72ba6fbe43236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:54.031886 19882 raft_consensus.cc:1261] T 0a4baa2ad3214b0c8fbc653fd415b24a P 54058034cc3742f7889db661e33e389e [term 1 FOLLOWER]: Refusing update from remote peer 96294b0dea84415792c72ba6fbe43236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:54.032261 20211 consensus_queue.cc:1035] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54058034cc3742f7889db661e33e389e" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 42575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:54.032410 20214 consensus_queue.cc:1035] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "651ed538c97641e6896952b6bf660b7c" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:54.035817 20219 mvcc.cc:205] Tried to move back new op lower bound from 7333495332991516672 to 7333495332907352064. Current Snapshot: MvccSnapshot[applied={T|T < 7333495332991516672}]
I20260926 06:28:54.037410 19739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:54.037501 19739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:54.038246 19739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c40b36d2bd41428fb8ff632f78cc6512: Sending DeleteTablet for 3 replicas of tablet 0a4baa2ad3214b0c8fbc653fd415b24a
I20260926 06:28:54.038556 20124 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a4baa2ad3214b0c8fbc653fd415b24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:54 UTC) from {username='slave'} at 127.0.0.1:34616
I20260926 06:28:54.038607 19862 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a4baa2ad3214b0c8fbc653fd415b24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:54 UTC) from {username='slave'} at 127.0.0.1:52736
I20260926 06:28:54.038604 19993 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a4baa2ad3214b0c8fbc653fd415b24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:54 UTC) from {username='slave'} at 127.0.0.1:54100
I20260926 06:28:54.038805 20226 tablet_replica.cc:330] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: stopping tablet replica
I20260926 06:28:54.038898 20226 raft_consensus.cc:2229] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:28:54.039018 20226 raft_consensus.cc:2258] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:54.039108 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 19800
I20260926 06:28:54.039450 20226 ts_tablet_manager.cc:1905] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:54.041445 20226 ts_tablet_manager.cc:1918] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:54.041531 20226 log.cc:1196] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/0a4baa2ad3214b0c8fbc653fd415b24a
I20260926 06:28:54.041807 20226 ts_tablet_manager.cc:1939] T 0a4baa2ad3214b0c8fbc653fd415b24a P 96294b0dea84415792c72ba6fbe43236: Deleting consensus metadata
I20260926 06:28:54.042261 19724 catalog_manager.cc:4841] TS 96294b0dea84415792c72ba6fbe43236 (127.2.39.195:42819): tablet 0a4baa2ad3214b0c8fbc653fd415b24a (table test-table [id=f8b9ea5047b94de3b11b49664751e2c4]) successfully deleted
I20260926 06:28:54.043056 20228 tablet_replica.cc:330] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: stopping tablet replica
I20260926 06:28:54.043140 20228 raft_consensus.cc:2229] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:54.043212 20228 raft_consensus.cc:2258] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:54.043608 20228 ts_tablet_manager.cc:1905] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:54.045187 20228 ts_tablet_manager.cc:1918] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:54.045303 20228 log.cc:1196] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/0a4baa2ad3214b0c8fbc653fd415b24a
I20260926 06:28:54.045655 20228 ts_tablet_manager.cc:1939] T 0a4baa2ad3214b0c8fbc653fd415b24a P 651ed538c97641e6896952b6bf660b7c: Deleting consensus metadata
I20260926 06:28:54.046119 19725 catalog_manager.cc:4841] TS 651ed538c97641e6896952b6bf660b7c (127.2.39.194:41833): tablet 0a4baa2ad3214b0c8fbc653fd415b24a (table test-table [id=f8b9ea5047b94de3b11b49664751e2c4]) successfully deleted
W20260926 06:28:54.046291 19725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:42575 (error 108)
I20260926 06:28:54.046774 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:54.046649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:42575 (local address 127.0.0.1:52744)
0926 06:28:54.046700 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:54.046741 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:42575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 06:28:54.046777 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 19931
W20260926 06:28:54.046881 19725 catalog_manager.cc:4584] TS 54058034cc3742f7889db661e33e389e (127.2.39.193:42575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a4baa2ad3214b0c8fbc653fd415b24a: Network error: Client connection negotiation failed: client connection to 127.2.39.193:42575: connect: Connection refused (error 111)
I20260926 06:28:54.052758 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 20062
I20260926 06:28:54.056571 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 19709
2026-09-26T06:28:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (786 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:28:54 dist-test-slave-fkbn krb5kdc[20232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:28:54 dist-test-slave-fkbn krb5kdc[20232](info): set up 2 sockets
Sep 26 06:28:54 dist-test-slave-fkbn krb5kdc[20232](info): commencing operation
krb5kdc: starting...
W20260926 06:28:55.109511 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:28:55 dist-test-slave-fkbn krb5kdc[20232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:28:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:28:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:28:55.226563 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:41503
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:41503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:55.306093 20245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:55.306303 20245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:55.306337 20245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:55.307500 20245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:28:55.307569 20245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:55.307593 20245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:55.307626 20245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:28:55.307650 20245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:28:55.309080 20245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:41503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:41503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:55.309342 20245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:55.309587 20245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:55.314419 20250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:55.314395 20251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:55.314389 20253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:55.314544 20245 server_base.cc:1034] running on GCE node
I20260926 06:28:55.314924 20245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:55.315227 20245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:55.316391 20245 hybrid_clock.cc:648] HybridClock initialized: now 1790404135316387 us; error 38 us; skew 500 ppm
Sep 26 06:28:55 dist-test-slave-fkbn krb5kdc[20232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:55.318794 20245 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:28:55.319208 20245 webserver.cc:460] Webserver started at http://127.2.39.254:33837/ using document root <none> and password file <none>
I20260926 06:28:55.319444 20245 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:55.319514 20245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:55.319664 20245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:55.320746 20245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "f9df38bd5e03c0864757c360ad6e4124"
server_key_iv: "d07b99d5a93e326daa66b408ff8e5c8c"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.321087 20245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "f9df38bd5e03c0864757c360ad6e4124"
server_key_iv: "d07b99d5a93e326daa66b408ff8e5c8c"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.322393 20245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:28:55.323055 20260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.323215 20245 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:28:55.323314 20245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "f9df38bd5e03c0864757c360ad6e4124"
server_key_iv: "d07b99d5a93e326daa66b408ff8e5c8c"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.323391 20245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:55.334265 20245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:55.335976 20245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:55.336135 20245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:55.339879 20245 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:41503
I20260926 06:28:55.339912 20312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:41503 every 8 connection(s)
I20260926 06:28:55.340276 20245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:28:55.340822 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 20245
I20260926 06:28:55.340939 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:28:55.341136 2207 external_mini_cluster.cc:1428] Setting key d3f512977429eaac6d7de94a87446b0e
I20260926 06:28:55.341221 20313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:55.344420 20313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6: Bootstrap starting.
I20260926 06:28:55.345094 20313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:55.345379 20313 log.cc:824] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:55.346127 20313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6: No bootstrap required, opened a new log
Sep 26 06:28:55 dist-test-slave-fkbn krb5kdc[20232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:55.348038 20313 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41503 } }
I20260926 06:28:55.348277 20313 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:55.348354 20313 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdb62f85a3ff4c3e8ff74ea9f5ec6fa6, State: Initialized, Role: FOLLOWER
I20260926 06:28:55.348486 20313 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41503 } }
I20260926 06:28:55.348556 20313 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:28:55.348609 20313 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:28:55.348667 20313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:55.349414 20313 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41503 } }
I20260926 06:28:55.349522 20313 leader_election.cc:302] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdb62f85a3ff4c3e8ff74ea9f5ec6fa6; no voters:
I20260926 06:28:55.349684 20313 leader_election.cc:288] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:28:55.349720 20318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:55.349876 20318 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [term 1 LEADER]: Becoming Leader. State: Replica: fdb62f85a3ff4c3e8ff74ea9f5ec6fa6, State: Running, Role: LEADER
I20260926 06:28:55.349946 20313 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:28:55.350008 20318 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41503 } }
I20260926 06:28:55.350584 20319 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41503 } } }
I20260926 06:28:55.350658 20319 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:55.350996 20327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:28:55.351575 20327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:28:55.351859 20320 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdb62f85a3ff4c3e8ff74ea9f5ec6fa6. Latest consensus state: current_term: 1 leader_uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdb62f85a3ff4c3e8ff74ea9f5ec6fa6" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 41503 } } }
I20260926 06:28:55.351931 20320 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:55.352370 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.341645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38986 (local address 127.2.39.254:41503)
0926 06:28:55.341835 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:55.341842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:55.341866 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:28:55.342340 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:55.342346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:55.342390 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:55.342511 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:55.343278 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.344108 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.345176 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.345709 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.345792 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.348016 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:55.348020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:55.348021 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:55.348033 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:55.350126 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:55.350532 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:55.350535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:55.350535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:55.350639 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:55.351064 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:55.351065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:55.351066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:55.351379 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:55.351558 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:55.352050 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 06:28:55.352178 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260926 06:28:55.353276 20327 catalog_manager.cc:1347] Generated new cluster ID: ff01757b77fd450e8ab0677f40742c7c
I20260926 06:28:55.353338 20327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:28:55.373395 20327 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:28:55.374133 20327 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:28:55.378253 20327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6: Generated new TSK 0
I20260926 06:28:55.378419 20327 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:28:55.404036 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41503
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:55.483665 20339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:55.483886 20339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:55.483922 20339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:55.485059 20339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:55.485122 20339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:55.485167 20339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:28:55.486652 20339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:55.486902 20339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:55.487124 20339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:55.492100 20347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:55.492087 20345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:55.492075 20344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:55.492422 20339 server_base.cc:1034] running on GCE node
I20260926 06:28:55.492606 20339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:55.492831 20339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:55.494009 20339 hybrid_clock.cc:648] HybridClock initialized: now 1790404135494003 us; error 35 us; skew 500 ppm
Sep 26 06:28:55 dist-test-slave-fkbn krb5kdc[20232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:55.496320 20339 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:28:55.496769 20339 webserver.cc:460] Webserver started at http://127.2.39.193:35959/ using document root <none> and password file <none>
I20260926 06:28:55.496991 20339 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:55.497057 20339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:55.497208 20339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:55.498528 20339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "8bac0c9be1b4485aae2ec9aa32a23828"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "ff0428f92ee482b8b499c851bb1c9a8d"
server_key_iv: "ab38054d0ef6ee8380b9947f4d1ef108"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.498927 20339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "8bac0c9be1b4485aae2ec9aa32a23828"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "ff0428f92ee482b8b499c851bb1c9a8d"
server_key_iv: "ab38054d0ef6ee8380b9947f4d1ef108"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.500962 20339 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 06:28:55.501686 20354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.501852 20339 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.501952 20339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "8bac0c9be1b4485aae2ec9aa32a23828"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "ff0428f92ee482b8b499c851bb1c9a8d"
server_key_iv: "ab38054d0ef6ee8380b9947f4d1ef108"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.502043 20339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:55.540263 20339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:55.542511 20339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:55.542678 20339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:55.542927 20339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:55.543248 20339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:55.543304 20339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.543344 20339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:55.543373 20339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.549165 20339 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:34639
I20260926 06:28:55.549229 20467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:34639 every 8 connection(s)
I20260926 06:28:55.549621 20339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
Sep 26 06:28:55 dist-test-slave-fkbn krb5kdc[20232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:55.558497 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.550341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:40113 (local address 127.2.39.254:41503)
0926 06:28:55.550412 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:55.550413 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:55.550683 (+ 270us) server_negotiation.cc:407] Connection header received
0926 06:28:55.551210 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:55.551212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:55.551223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:55.551276 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:55.552287 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.552794 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.553862 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.554093 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.554171 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.556177 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:55.556179 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:55.556179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:55.556185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:55.557350 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:55.557771 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:55.557772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:55.557772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:55.557823 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:55.558017 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:55.558018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:55.558018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:55.558111 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:55.558157 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:55.558376 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 06:28:55.558435 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:28:55.558673 20470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.550330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41503 (local address 127.2.39.193:40113)
0926 06:28:55.550563 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:55.550598 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:55.551106 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:55.551357 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:55.551360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:55.551981 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:55.552175 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.552180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.552903 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.552904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.553765 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.553767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.554176 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.554177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.554308 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.554728 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:55.554736 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:55.556093 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:55.557418 (+ 1325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:55.557424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:55.557425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:55.557709 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:55.557882 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:55.557882 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:55.557883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:55.557977 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:55.558160 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:55.558164 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:55.558275 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 06:28:55.558354 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 06:28:55.558478 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":110,"threads_started":1}
I20260926 06:28:55.559207 20468 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41503
I20260926 06:28:55.559314 20468 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:55.559334 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 20339
I20260926 06:28:55.559465 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:28:55.559540 20468 heartbeater.cc:511] Master 127.2.39.254:41503 requested a full tablet report, sending...
I20260926 06:28:55.559666 2207 external_mini_cluster.cc:1428] Setting key d52e02d304cea8929eb3e27b9136b0a7
I20260926 06:28:55.560060 20277 ts_manager.cc:194] Registered new tserver with Master: 8bac0c9be1b4485aae2ec9aa32a23828 (127.2.39.193:34639)
I20260926 06:28:55.560811 20277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:40113
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:28:55.583917 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41503
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:55.674418 20473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:55.674610 20473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:55.674633 20473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:55.675705 20473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:55.675741 20473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:55.675766 20473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:28:55.677099 20473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:55.677271 20473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:55.677433 20473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:55.682421 20481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:55.682397 20479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:55.682397 20478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:55.682668 20473 server_base.cc:1034] running on GCE node
I20260926 06:28:55.682853 20473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:55.683091 20473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:55.684247 20473 hybrid_clock.cc:648] HybridClock initialized: now 1790404135684225 us; error 34 us; skew 500 ppm
Sep 26 06:28:55 dist-test-slave-fkbn krb5kdc[20232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:55.686815 20473 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:28:55.687265 20473 webserver.cc:460] Webserver started at http://127.2.39.194:32901/ using document root <none> and password file <none>
I20260926 06:28:55.687497 20473 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:55.687563 20473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:55.687711 20473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:55.688709 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "9980885b45d04562a669867e2098b711"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "89c2fc2d42b0700ee2f8d6a82d067b7f"
server_key_iv: "4711b5276680b46aca9eb6744888b01e"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.689060 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "9980885b45d04562a669867e2098b711"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "89c2fc2d42b0700ee2f8d6a82d067b7f"
server_key_iv: "4711b5276680b46aca9eb6744888b01e"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.690514 20473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:28:55.691318 20488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.691466 20473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.691565 20473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "9980885b45d04562a669867e2098b711"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "89c2fc2d42b0700ee2f8d6a82d067b7f"
server_key_iv: "4711b5276680b46aca9eb6744888b01e"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.691639 20473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:55.700913 20473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:55.703151 20473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:55.703305 20473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:55.703526 20473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:55.703806 20473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:55.703860 20473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.703899 20473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:55.703927 20473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.709724 20473 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:44221
I20260926 06:28:55.709761 20601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:44221 every 8 connection(s)
I20260926 06:28:55.710078 20473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:28:55.710420 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 20473
I20260926 06:28:55.710554 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:28:55.710758 2207 external_mini_cluster.cc:1428] Setting key a3e8d607689a5a24c8d2fc82072c5155
Sep 26 06:28:55 dist-test-slave-fkbn krb5kdc[20232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
I20260926 06:28:55.719892 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.710779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:41473 (local address 127.2.39.254:41503)
0926 06:28:55.710842 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:55.710844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:55.711017 (+ 173us) server_negotiation.cc:407] Connection header received
0926 06:28:55.711517 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:55.711517 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:55.711526 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:55.711559 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:55.712776 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.713289 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.714391 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.714589 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.714673 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.717053 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:55.717054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:55.717055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:55.717062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:55.718515 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:55.719022 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:55.719022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:55.719023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:55.719072 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:55.719277 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:55.719277 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:55.719278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:55.719381 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:55.719430 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:55.719764 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 06:28:55.719827 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 06:28:55.720043 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.710715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41503 (local address 127.2.39.194:41473)
0926 06:28:55.710881 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:55.710919 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:55.711394 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:55.711641 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:55.711645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:55.712435 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:55.712695 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.712701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.713386 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.713387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.714289 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.714292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.714680 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.714681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.714814 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.715372 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:55.715380 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:55.716957 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:55.718658 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:55.718664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:55.718667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:55.718954 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:55.719131 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:55.719132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:55.719133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:55.719224 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:55.719435 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:55.719449 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:55.719610 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 06:28:55.719724 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 06:28:55.719856 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260926 06:28:55.720494 20602 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41503
I20260926 06:28:55.720597 20602 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:55.720767 20602 heartbeater.cc:511] Master 127.2.39.254:41503 requested a full tablet report, sending...
I20260926 06:28:55.721148 20277 ts_manager.cc:194] Registered new tserver with Master: 9980885b45d04562a669867e2098b711 (127.2.39.194:44221)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
I20260926 06:28:55.721544 20277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:41473
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:28:55.734591 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41503
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:55.812842 20607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:55.813057 20607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:55.813094 20607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:55.814234 20607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:55.814301 20607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:55.814344 20607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:28:55.815784 20607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:41503
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:55.816040 20607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:55.816262 20607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:55.821105 20615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:55.821146 20607 server_base.cc:1034] running on GCE node
W20260926 06:28:55.821099 20613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:55.821099 20612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:55.821472 20607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:55.821791 20607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:55.822947 20607 hybrid_clock.cc:648] HybridClock initialized: now 1790404135822936 us; error 40 us; skew 500 ppm
Sep 26 06:28:55 dist-test-slave-fkbn krb5kdc[20232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:55.825235 20607 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:28:55.825727 20607 webserver.cc:460] Webserver started at http://127.2.39.195:37147/ using document root <none> and password file <none>
I20260926 06:28:55.825973 20607 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:55.826041 20607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:55.826186 20607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:55.827162 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "c20c4801a6134bb5a41c3e2320dd29e0"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "3b02ed2388a4f7a07ede0a94fc171ea8"
server_key_iv: "6668c51a3f5a969f2bd2b9727780109c"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.827513 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "c20c4801a6134bb5a41c3e2320dd29e0"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "3b02ed2388a4f7a07ede0a94fc171ea8"
server_key_iv: "6668c51a3f5a969f2bd2b9727780109c"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.828738 20607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:28:55.829437 20622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.829874 20607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:28:55.829972 20607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "c20c4801a6134bb5a41c3e2320dd29e0"
format_stamp: "Formatted at 2026-09-26 06:28:55 on dist-test-slave-fkbn"
server_key: "3b02ed2388a4f7a07ede0a94fc171ea8"
server_key_iv: "6668c51a3f5a969f2bd2b9727780109c"
server_key_version: "encryptionkey@0"
I20260926 06:28:55.830054 20607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:55.871064 20607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:55.872910 20607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:55.873076 20607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:55.873324 20607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:55.873698 20607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:55.873754 20607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.873793 20607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:55.873821 20607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:55.879595 20607 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:40987
I20260926 06:28:55.879635 20735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:40987 every 8 connection(s)
I20260926 06:28:55.879978 20607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:28:55 dist-test-slave-fkbn krb5kdc[20232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:55.888232 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.880690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:57279 (local address 127.2.39.254:41503)
0926 06:28:55.880756 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:55.880758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:55.880924 (+ 166us) server_negotiation.cc:407] Connection header received
0926 06:28:55.881387 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:55.881388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:55.881396 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:55.881453 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:55.882391 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.882877 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.883832 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.884020 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.884069 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.885984 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:55.885985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:55.885986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:55.885992 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:55.887135 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:55.887493 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:55.887494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:55.887494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:55.887543 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:55.887742 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:55.887743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:55.887743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:55.887837 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:55.887887 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:55.888111 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 06:28:55.888171 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:28:55.888422 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.880651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:41503 (local address 127.2.39.195:57279)
0926 06:28:55.880800 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:55.880840 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:55.881308 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:55.881504 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:55.881507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:55.882126 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:55.882311 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.882315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.882964 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.882965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.883757 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.883759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.884087 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.884088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.884179 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.884584 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:55.884592 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:55.885901 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:55.887203 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:55.887209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:55.887211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:55.887413 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:55.887599 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:55.887599 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:55.887599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:55.887689 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:55.887891 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:55.887895 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:55.888004 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 06:28:55.888084 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 06:28:55.888194 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260926 06:28:55.888921 20736 heartbeater.cc:347] Connected to a master server at 127.2.39.254:41503
I20260926 06:28:55.889026 20736 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:55.889214 20736 heartbeater.cc:511] Master 127.2.39.254:41503 requested a full tablet report, sending...
I20260926 06:28:55.889343 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 20607
I20260926 06:28:55.889448 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:28:55.889626 20277 ts_manager.cc:194] Registered new tserver with Master: c20c4801a6134bb5a41c3e2320dd29e0 (127.2.39.195:40987)
I20260926 06:28:55.889655 2207 external_mini_cluster.cc:1428] Setting key 1128c709a28edd8a54f420bed63d3482
I20260926 06:28:55.890079 20277 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:57279
I20260926 06:28:55.890563 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:28:55.897753 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.891880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.2.39.254:41503)
0926 06:28:55.891934 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:55.891936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:55.892058 (+ 122us) server_negotiation.cc:407] Connection header received
0926 06:28:55.892122 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:55.892122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:55.892132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:55.892185 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:55.892698 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.893152 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.893925 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.894142 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.894242 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.895356 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:55.895358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:55.895358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:55.895365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:55.896518 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:55.896883 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:55.896884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:55.896884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:55.896939 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:55.897158 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:55.897159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:55.897160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:55.897333 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:55.897457 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:55.897597 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 06:28:55.897658 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:28:55.899060 20277 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:28:55.900506 20277 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:28:55.906826 20748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.902838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:40987 (local address 127.0.0.1:49620)
0926 06:28:55.902964 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:55.902975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:55.903028 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:55.903322 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:55.903325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:55.903327 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:55.903394 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.903395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.904699 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.904700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.905925 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.905927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.906540 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.906541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.906626 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.906638 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 06:28:55.906711 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 06:28:55.906759 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260926 06:28:55.906872 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.903008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49620 (local address 127.2.39.195:40987)
0926 06:28:55.903162 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:55.903167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:55.903176 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:28:55.903209 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:55.903211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:55.903225 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:55.903304 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:55.903492 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.904596 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.906006 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.906438 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.906538 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.906595 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:55.906745 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 06:28:55.906795 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
I20260926 06:28:55.907882 20751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.902640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:34639 (local address 127.0.0.1:35988)
0926 06:28:55.903101 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:55.903110 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:55.903156 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:55.903535 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:55.903536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:55.903538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:55.903582 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.903583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.904567 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.904568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.905924 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.905927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.907588 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.907589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.907721 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.907724 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:55.907770 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 06:28:55.907809 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":5,"thread_start_us":76,"threads_started":1}
I20260926 06:28:55.907852 20670 tablet_service.cc:1467] Processing CreateTablet for tablet baab0d2a191f45f59852232d51cdc3e1 (DEFAULT_TABLE table=test-table [id=4b732b7ad362424b84e39d3115ea0719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:55.908025 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.902638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:44221 (local address 127.0.0.1:47494)
0926 06:28:55.902909 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:55.902964 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:55.903027 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:55.903322 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:55.903325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:55.903329 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:55.903394 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.903395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.906024 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.906025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.906711 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:55.906713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.907632 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.907633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:55.907710 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.907714 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:55.907768 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 06:28:55.907809 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":8,"thread_start_us":64,"threads_started":1}
I20260926 06:28:55.908216 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.902751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.2.39.193:34639)
0926 06:28:55.902886 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:55.902891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:55.903168 (+ 277us) server_negotiation.cc:407] Connection header received
0926 06:28:55.903228 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:55.903230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:55.903247 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:55.903458 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:55.903651 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.904460 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.906629 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.907394 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.907460 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.907522 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:55.908097 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 06:28:55.908146 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260926 06:28:55.908232 20670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baab0d2a191f45f59852232d51cdc3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:55.908401 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:55.902735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.2.39.194:44221)
0926 06:28:55.902985 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:55.902991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:55.903005 (+ 14us) server_negotiation.cc:407] Connection header received
0926 06:28:55.903124 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:55.903126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:55.903143 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:55.903214 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:55.903480 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.904596 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.906803 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:55.907489 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:55.907558 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:55.907610 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:55.907853 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 06:28:55.907897 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
I20260926 06:28:55.908849 20402 tablet_service.cc:1467] Processing CreateTablet for tablet baab0d2a191f45f59852232d51cdc3e1 (DEFAULT_TABLE table=test-table [id=4b732b7ad362424b84e39d3115ea0719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:55.909159 20402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baab0d2a191f45f59852232d51cdc3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:55.911581 20753 tablet_bootstrap.cc:492] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: Bootstrap starting.
I20260926 06:28:55.912186 20753 tablet_bootstrap.cc:654] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:55.912231 20536 tablet_service.cc:1467] Processing CreateTablet for tablet baab0d2a191f45f59852232d51cdc3e1 (DEFAULT_TABLE table=test-table [id=4b732b7ad362424b84e39d3115ea0719]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:55.912550 20536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baab0d2a191f45f59852232d51cdc3e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:55.912560 20754 tablet_bootstrap.cc:492] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828: Bootstrap starting.
I20260926 06:28:55.912808 20753 log.cc:824] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:55.913192 20754 tablet_bootstrap.cc:654] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:55.913663 20753 tablet_bootstrap.cc:492] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: No bootstrap required, opened a new log
I20260926 06:28:55.913738 20753 ts_tablet_manager.cc:1397] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:28:55.913796 20754 log.cc:824] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:55.915153 20753 raft_consensus.cc:348] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9980885b45d04562a669867e2098b711" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44221 } } peers { permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 } } peers { permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 } }
I20260926 06:28:55.915339 20753 raft_consensus.cc:374] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:55.915354 20757 tablet_bootstrap.cc:492] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: Bootstrap starting.
I20260926 06:28:55.915390 20753 raft_consensus.cc:729] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20c4801a6134bb5a41c3e2320dd29e0, State: Initialized, Role: FOLLOWER
I20260926 06:28:55.915519 20753 consensus_queue.cc:260] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9980885b45d04562a669867e2098b711" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44221 } } peers { permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 } } peers { permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 } }
I20260926 06:28:55.915815 20753 ts_tablet_manager.cc:1428] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:28:55.915843 20736 heartbeater.cc:503] Master 127.2.39.254:41503 was elected leader, sending a full tablet report...
I20260926 06:28:55.915995 20757 tablet_bootstrap.cc:654] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:55.916453 20754 tablet_bootstrap.cc:492] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828: No bootstrap required, opened a new log
I20260926 06:28:55.916487 20757 log.cc:824] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:55.916523 20754 ts_tablet_manager.cc:1397] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 06:28:55.917177 20757 tablet_bootstrap.cc:492] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: No bootstrap required, opened a new log
I20260926 06:28:55.917228 20757 ts_tablet_manager.cc:1397] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:28:55.918231 20754 raft_consensus.cc:348] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9980885b45d04562a669867e2098b711" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44221 } } peers { permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 } } peers { permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 } }
I20260926 06:28:55.918447 20754 raft_consensus.cc:374] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:55.918495 20754 raft_consensus.cc:729] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bac0c9be1b4485aae2ec9aa32a23828, State: Initialized, Role: FOLLOWER
I20260926 06:28:55.918612 20754 consensus_queue.cc:260] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9980885b45d04562a669867e2098b711" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44221 } } peers { permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 } } peers { permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 } }
I20260926 06:28:55.918891 20754 ts_tablet_manager.cc:1428] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:28:55.918923 20468 heartbeater.cc:503] Master 127.2.39.254:41503 was elected leader, sending a full tablet report...
I20260926 06:28:55.918895 20757 raft_consensus.cc:348] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9980885b45d04562a669867e2098b711" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44221 } } peers { permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 } } peers { permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 } }
I20260926 06:28:55.919035 20757 raft_consensus.cc:374] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:55.919059 20757 raft_consensus.cc:729] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9980885b45d04562a669867e2098b711, State: Initialized, Role: FOLLOWER
I20260926 06:28:55.919181 20757 consensus_queue.cc:260] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9980885b45d04562a669867e2098b711" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44221 } } peers { permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 } } peers { permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 } }
I20260926 06:28:55.919461 20757 ts_tablet_manager.cc:1428] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:28:55.919479 20602 heartbeater.cc:503] Master 127.2.39.254:41503 was elected leader, sending a full tablet report...
W20260926 06:28:55.960676 20603 tablet.cc:2369] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 06:28:56.050372 20469 tablet.cc:2369] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:28:56.088804 20761 raft_consensus.cc:482] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:56.088960 20761 raft_consensus.cc:504] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9980885b45d04562a669867e2098b711" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44221 } } peers { permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 } } peers { permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 } }
I20260926 06:28:56.089360 20761 leader_election.cc:288] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bac0c9be1b4485aae2ec9aa32a23828 (127.2.39.193:34639), c20c4801a6134bb5a41c3e2320dd29e0 (127.2.39.195:40987)
I20260926 06:28:56.093011 20749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.089651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:38769 (local address 127.2.39.193:34639)
0926 06:28:56.089695 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.089697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.089992 (+ 295us) server_negotiation.cc:407] Connection header received
0926 06:28:56.090054 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.090055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.090068 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.090116 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:56.090283 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.091004 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.092211 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.092653 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.092728 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.092736 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.092905 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.092942 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:28:56.093057 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.089531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:34639 (local address 127.2.39.194:38769)
0926 06:28:56.089901 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.089913 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.089991 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.090131 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.090132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.090133 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:56.090212 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.090214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.091091 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.091092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.092126 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.092129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.092738 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.092740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.092823 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.092829 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.092892 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.092938 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":194,"thread_start_us":103,"threads_started":1}
I20260926 06:28:56.093430 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.089529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:40987 (local address 127.2.39.194:36949)
0926 06:28:56.089611 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.089624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.089673 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.089921 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.089921 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.089923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:56.090025 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.090027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.091006 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.091008 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.092404 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.092406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.093177 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.093178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.093275 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.093278 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.093324 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.093364 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":51}
I20260926 06:28:56.093542 20422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baab0d2a191f45f59852232d51cdc3e1" candidate_uuid: "9980885b45d04562a669867e2098b711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" is_pre_election: true
I20260926 06:28:56.093751 20752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.089695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:36949 (local address 127.2.39.195:40987)
0926 06:28:56.089766 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.089768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.089777 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:28:56.089804 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.089805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.089814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.089868 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:56.090116 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.090853 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.092500 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.093064 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.093135 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.093143 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.093640 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.093680 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:28:56.093761 20422 raft_consensus.cc:2454] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9980885b45d04562a669867e2098b711 in term 0.
I20260926 06:28:56.094010 20492 leader_election.cc:302] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bac0c9be1b4485aae2ec9aa32a23828, 9980885b45d04562a669867e2098b711; no voters:
I20260926 06:28:56.094038 20690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baab0d2a191f45f59852232d51cdc3e1" candidate_uuid: "9980885b45d04562a669867e2098b711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" is_pre_election: true
I20260926 06:28:56.094137 20761 raft_consensus.cc:2781] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:28:56.094182 20690 raft_consensus.cc:2454] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9980885b45d04562a669867e2098b711 in term 0.
I20260926 06:28:56.094192 20761 raft_consensus.cc:482] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:28:56.094229 20761 raft_consensus.cc:3037] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:56.095108 20761 raft_consensus.cc:504] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9980885b45d04562a669867e2098b711" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44221 } } peers { permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 } } peers { permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 } }
I20260926 06:28:56.095264 20761 leader_election.cc:288] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [CANDIDATE]: Term 1 election: Requested vote from peers 8bac0c9be1b4485aae2ec9aa32a23828 (127.2.39.193:34639), c20c4801a6134bb5a41c3e2320dd29e0 (127.2.39.195:40987)
I20260926 06:28:56.095387 20690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baab0d2a191f45f59852232d51cdc3e1" candidate_uuid: "9980885b45d04562a669867e2098b711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c20c4801a6134bb5a41c3e2320dd29e0"
I20260926 06:28:56.095417 20422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baab0d2a191f45f59852232d51cdc3e1" candidate_uuid: "9980885b45d04562a669867e2098b711" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bac0c9be1b4485aae2ec9aa32a23828"
I20260926 06:28:56.095486 20690 raft_consensus.cc:3037] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:56.095496 20422 raft_consensus.cc:3037] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:56.096246 20690 raft_consensus.cc:2454] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9980885b45d04562a669867e2098b711 in term 1.
I20260926 06:28:56.096414 20489 leader_election.cc:302] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9980885b45d04562a669867e2098b711, c20c4801a6134bb5a41c3e2320dd29e0; no voters:
I20260926 06:28:56.096524 20761 raft_consensus.cc:2781] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:56.096554 20422 raft_consensus.cc:2454] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9980885b45d04562a669867e2098b711 in term 1.
I20260926 06:28:56.096673 20761 raft_consensus.cc:686] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 1 LEADER]: Becoming Leader. State: Replica: 9980885b45d04562a669867e2098b711, State: Running, Role: LEADER
I20260926 06:28:56.096772 20761 consensus_queue.cc:237] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9980885b45d04562a669867e2098b711" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44221 } } peers { permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 } } peers { permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 } }
I20260926 06:28:56.097683 20277 catalog_manager.cc:5495] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9980885b45d04562a669867e2098b711 (127.2.39.194). New cstate: current_term: 1 leader_uuid: "9980885b45d04562a669867e2098b711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9980885b45d04562a669867e2098b711" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 44221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 } health_report { overall_health: UNKNOWN } } }
W20260926 06:28:56.130525 20737 tablet.cc:2369] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:28:56.132794 20750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.130513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47500 (local address 127.2.39.194:44221)
0926 06:28:56.130579 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.130582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.130590 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:28:56.130644 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.130645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.130669 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.130739 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:28:56.130906 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.131362 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.132078 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.132270 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.132317 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.132469 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:28:56.132557 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:28:56.132673 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.132701 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.132713 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:28:56.134635 20422 raft_consensus.cc:1261] T baab0d2a191f45f59852232d51cdc3e1 P 8bac0c9be1b4485aae2ec9aa32a23828 [term 1 FOLLOWER]: Refusing update from remote peer 9980885b45d04562a669867e2098b711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:56.134660 20690 raft_consensus.cc:1261] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0 [term 1 FOLLOWER]: Refusing update from remote peer 9980885b45d04562a669867e2098b711: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:56.135001 20763 consensus_queue.cc:1035] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bac0c9be1b4485aae2ec9aa32a23828" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 34639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:56.135149 20761 consensus_queue.cc:1035] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c20c4801a6134bb5a41c3e2320dd29e0" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 40987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:56.137318 20766 mvcc.cc:205] Tried to move back new op lower bound from 7333495341604712448 to 7333495341452939264. Current Snapshot: MvccSnapshot[applied={T|T < 7333495341604712448}]
I20260926 06:28:56.137723 20767 mvcc.cc:205] Tried to move back new op lower bound from 7333495341604712448 to 7333495341452939264. Current Snapshot: MvccSnapshot[applied={T|T < 7333495341604712448}]
I20260926 06:28:56.139987 20277 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:56.140081 20277 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:56.140776 20277 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdb62f85a3ff4c3e8ff74ea9f5ec6fa6: Sending DeleteTablet for 3 replicas of tablet baab0d2a191f45f59852232d51cdc3e1
I20260926 06:28:56.141142 20402 tablet_service.cc:1514] Processing DeleteTablet for tablet baab0d2a191f45f59852232d51cdc3e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:56 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:35988
I20260926 06:28:56.141192 20536 tablet_service.cc:1514] Processing DeleteTablet for tablet baab0d2a191f45f59852232d51cdc3e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:56 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:47494
I20260926 06:28:56.141214 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet baab0d2a191f45f59852232d51cdc3e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:56 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:49620
I20260926 06:28:56.141438 20778 tablet_replica.cc:330] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: stopping tablet replica
I20260926 06:28:56.141536 20778 raft_consensus.cc:2229] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:56.141556 20776 tablet_replica.cc:330] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: stopping tablet replica
I20260926 06:28:56.141666 20778 raft_consensus.cc:2258] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:56.141713 20776 raft_consensus.cc:2229] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:28:56.141785 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 20339
I20260926 06:28:56.141865 20776 raft_consensus.cc:2258] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:56.142709 20776 ts_tablet_manager.cc:1905] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:56.144430 20778 ts_tablet_manager.cc:1905] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:56.144815 20776 ts_tablet_manager.cc:1918] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:56.144936 20776 log.cc:1196] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/baab0d2a191f45f59852232d51cdc3e1
I20260926 06:28:56.145256 20776 ts_tablet_manager.cc:1939] T baab0d2a191f45f59852232d51cdc3e1 P 9980885b45d04562a669867e2098b711: Deleting consensus metadata
I20260926 06:28:56.145709 20261 catalog_manager.cc:4841] TS 9980885b45d04562a669867e2098b711 (127.2.39.194:44221): tablet baab0d2a191f45f59852232d51cdc3e1 (table test-table [id=4b732b7ad362424b84e39d3115ea0719]) successfully deleted
I20260926 06:28:56.146320 20778 ts_tablet_manager.cc:1918] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:56.146395 20778 log.cc:1196] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/baab0d2a191f45f59852232d51cdc3e1
I20260926 06:28:56.146687 20778 ts_tablet_manager.cc:1939] T baab0d2a191f45f59852232d51cdc3e1 P c20c4801a6134bb5a41c3e2320dd29e0: Deleting consensus metadata
I20260926 06:28:56.147070 20261 catalog_manager.cc:4841] TS c20c4801a6134bb5a41c3e2320dd29e0 (127.2.39.195:40987): tablet baab0d2a191f45f59852232d51cdc3e1 (table test-table [id=4b732b7ad362424b84e39d3115ea0719]) successfully deleted
W20260926 06:28:56.148151 20264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:34639 (error 108)
I20260926 06:28:56.148677 20747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.148557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:34639 (local address 127.0.0.1:35992)
0926 06:28:56.148607 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.148645 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:34639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 06:28:56.148826 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 20473
W20260926 06:28:56.148819 20264 catalog_manager.cc:4584] TS 8bac0c9be1b4485aae2ec9aa32a23828 (127.2.39.193:34639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet baab0d2a191f45f59852232d51cdc3e1: Network error: Client connection negotiation failed: client connection to 127.2.39.193:34639: connect: Connection refused (error 111)
I20260926 06:28:56.153187 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 20607
I20260926 06:28:56.158241 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 20245
2026-09-26T06:28:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2103 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T06:28:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:28:56Z Disabled control of system clock
I20260926 06:28:56.185386 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:40397
--webserver_interface=127.2.39.254
--webserver_port=0
--builtin_ntp_servers=127.2.39.212:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:28:56.264669 20783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:56.264874 20783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:56.264894 20783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:56.266093 20783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:28:56.266132 20783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:56.266145 20783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:56.266158 20783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:28:56.266203 20783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:28:56.267480 20783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:40397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:56.267717 20783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:56.268000 20783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:56.272761 20791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:56.272701 20788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:56.272701 20789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:56.273044 20783 server_base.cc:1034] running on GCE node
I20260926 06:28:56.273269 20783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:56.273537 20783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:56.274722 20783 hybrid_clock.cc:648] HybridClock initialized: now 1790404136274710 us; error 28 us; skew 500 ppm
I20260926 06:28:56.275964 20783 webserver.cc:460] Webserver started at http://127.2.39.254:36129/ using document root <none> and password file <none>
I20260926 06:28:56.276214 20783 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:56.276261 20783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:56.276418 20783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:56.277514 20783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "53c6853d997c4fafb20984119839987c"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "ce061426c6c8233c121c43684dfc32db"
server_key_iv: "b64459abbfed4db568a876725637b8fc"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.277930 20783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "53c6853d997c4fafb20984119839987c"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "ce061426c6c8233c121c43684dfc32db"
server_key_iv: "b64459abbfed4db568a876725637b8fc"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.279227 20783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 06:28:56.279901 20797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:56.280074 20783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:28:56.280143 20783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "53c6853d997c4fafb20984119839987c"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "ce061426c6c8233c121c43684dfc32db"
server_key_iv: "b64459abbfed4db568a876725637b8fc"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.280229 20783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:56.306046 20783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:56.306372 20783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:56.306528 20783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:56.310362 20783 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:40397
I20260926 06:28:56.310400 20849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:40397 every 8 connection(s)
I20260926 06:28:56.310738 20783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:28:56.311730 20850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:56.314069 20850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c: Bootstrap starting.
I20260926 06:28:56.314662 20850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:56.314977 20850 log.cc:824] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:56.315737 20850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c: No bootstrap required, opened a new log
I20260926 06:28:56.317399 20850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6853d997c4fafb20984119839987c" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 40397 } }
I20260926 06:28:56.317539 20850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:56.317561 20850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c6853d997c4fafb20984119839987c, State: Initialized, Role: FOLLOWER
I20260926 06:28:56.317802 20850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6853d997c4fafb20984119839987c" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 40397 } }
I20260926 06:28:56.317871 20850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:28:56.317898 20850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:28:56.317945 20850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:56.318454 20850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6853d997c4fafb20984119839987c" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 40397 } }
I20260926 06:28:56.318534 20850 leader_election.cc:302] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53c6853d997c4fafb20984119839987c; no voters:
I20260926 06:28:56.318745 20850 leader_election.cc:288] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:28:56.318787 20853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:56.318965 20853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [term 1 LEADER]: Becoming Leader. State: Replica: 53c6853d997c4fafb20984119839987c, State: Running, Role: LEADER
I20260926 06:28:56.319031 20850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:28:56.319079 20853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6853d997c4fafb20984119839987c" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 40397 } }
I20260926 06:28:56.319550 20854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53c6853d997c4fafb20984119839987c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6853d997c4fafb20984119839987c" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 40397 } } }
I20260926 06:28:56.319646 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 20783
I20260926 06:28:56.319761 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:28:56.319751 20854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:56.319983 2207 external_mini_cluster.cc:1428] Setting key e42c3e0cece209163836694267d618f1
I20260926 06:28:56.320078 20855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53c6853d997c4fafb20984119839987c. Latest consensus state: current_term: 1 leader_uuid: "53c6853d997c4fafb20984119839987c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53c6853d997c4fafb20984119839987c" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 40397 } } }
I20260926 06:28:56.320145 20855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:56.320262 20863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:28:56.321028 20863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:28:56.322588 20863 catalog_manager.cc:1347] Generated new cluster ID: c671be28e0dc4218bc78f6480d7fa054
I20260926 06:28:56.322640 20863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:28:56.327593 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.321667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.2.39.254:40397)
0926 06:28:56.321875 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.321883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.321913 (+ 30us) server_negotiation.cc:407] Connection header received
0926 06:28:56.322035 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.322041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.322093 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.322279 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:56.322546 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.323897 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.325040 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.325678 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.326566 (+ 888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.326717 (+ 151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:56.326720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:56.326721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:56.326919 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:56.326956 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:56.326970 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:56.327261 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.327296 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.327370 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260926 06:28:56.331251 20863 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:28:56.331940 20863 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:28:56.338877 20863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c: Generated new TSK 0
I20260926 06:28:56.339074 20863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:28:56.342720 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:40397
--builtin_ntp_servers=127.2.39.212:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:28:56.427229 20874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:56.427459 20874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:56.427497 20874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:56.428632 20874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:56.428694 20874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:56.428735 20874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:28:56.430080 20874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:56.430326 20874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:56.430610 20874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:56.436475 20882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:56.436420 20880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:56.436633 20874 server_base.cc:1034] running on GCE node
W20260926 06:28:56.436443 20879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:56.436915 20874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:56.437170 20874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:56.438375 20874 hybrid_clock.cc:648] HybridClock initialized: now 1790404136438350 us; error 39 us; skew 500 ppm
I20260926 06:28:56.439689 20874 webserver.cc:460] Webserver started at http://127.2.39.193:37323/ using document root <none> and password file <none>
I20260926 06:28:56.439973 20874 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:56.440045 20874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:56.440186 20874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:56.441222 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "7365e88f7feb4f3fa28855b2478f7511"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "68e582365b23556a1a4c17128920a963"
server_key_iv: "77e6446b10258245c41f61c11139cbe9"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.441639 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "7365e88f7feb4f3fa28855b2478f7511"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "68e582365b23556a1a4c17128920a963"
server_key_iv: "77e6446b10258245c41f61c11139cbe9"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.442989 20874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:28:56.443817 20888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:56.443998 20874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:28:56.444098 20874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "7365e88f7feb4f3fa28855b2478f7511"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "68e582365b23556a1a4c17128920a963"
server_key_iv: "77e6446b10258245c41f61c11139cbe9"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.444185 20874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:56.460675 20874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:56.460996 20874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:56.461151 20874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:56.461382 20874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:56.461817 20874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:56.461874 20874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:56.461917 20874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:56.461953 20874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:56.467427 20874 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:40375
I20260926 06:28:56.467519 21001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:40375 every 8 connection(s)
I20260926 06:28:56.467819 20874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:28:56.468372 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 20874
I20260926 06:28:56.468493 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:28:56.468717 2207 external_mini_cluster.cc:1428] Setting key 42cfa81c71097f4030663d38a30a8349
I20260926 06:28:56.469712 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:40397
--builtin_ntp_servers=127.2.39.212:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 06:28:56.473486 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.468550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:34801 (local address 127.2.39.254:40397)
0926 06:28:56.468603 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.468605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.468878 (+ 273us) server_negotiation.cc:407] Connection header received
0926 06:28:56.469556 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.469556 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.469602 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.469652 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:56.470534 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.471106 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.472055 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.472285 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.472341 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.473245 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:56.473246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:56.473247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:56.473250 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:56.473262 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:56.473266 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:56.473307 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.473387 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.473416 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:28:56.473799 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.468555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:40397 (local address 127.2.39.193:34801)
0926 06:28:56.468741 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.468788 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.469446 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.469728 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.469732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.469743 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:56.470443 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.470450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.471206 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.471207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.471969 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.471971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.472381 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.472381 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.472552 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.473093 (+ 541us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:28:56.473093 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:56.473132 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:28:56.473160 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:56.473310 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:56.473314 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:56.473318 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.473390 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.473514 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260926 06:28:56.474315 21002 heartbeater.cc:347] Connected to a master server at 127.2.39.254:40397
I20260926 06:28:56.474416 21002 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:56.474632 21002 heartbeater.cc:511] Master 127.2.39.254:40397 requested a full tablet report, sending...
I20260926 06:28:56.475140 20814 ts_manager.cc:194] Registered new tserver with Master: 7365e88f7feb4f3fa28855b2478f7511 (127.2.39.193:40375)
I20260926 06:28:56.475749 20814 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.193:34801
W20260926 06:28:56.552109 21005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:56.552302 21005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:56.552323 21005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:56.553486 21005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:56.553529 21005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:56.553553 21005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:28:56.554996 21005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:56.555253 21005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:56.555549 21005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:56.560374 21011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:56.560398 21010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:56.560407 21013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:56.560593 21005 server_base.cc:1034] running on GCE node
I20260926 06:28:56.560768 21005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:56.560956 21005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:56.562098 21005 hybrid_clock.cc:648] HybridClock initialized: now 1790404136562084 us; error 31 us; skew 500 ppm
I20260926 06:28:56.563233 21005 webserver.cc:460] Webserver started at http://127.2.39.194:37207/ using document root <none> and password file <none>
I20260926 06:28:56.563479 21005 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:56.563526 21005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:56.563673 21005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:56.564730 21005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "e3aa447cafc54bac8587cad592ab0c0e"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "bd64b5b495223559137a6974bb80033a"
server_key_iv: "c7a91c45fb2f4b4978dbc54c32d9b289"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.565089 21005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "e3aa447cafc54bac8587cad592ab0c0e"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "bd64b5b495223559137a6974bb80033a"
server_key_iv: "c7a91c45fb2f4b4978dbc54c32d9b289"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.566433 21005 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:28:56.567112 21019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:56.567296 21005 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:28:56.567363 21005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "e3aa447cafc54bac8587cad592ab0c0e"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "bd64b5b495223559137a6974bb80033a"
server_key_iv: "c7a91c45fb2f4b4978dbc54c32d9b289"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.567451 21005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:56.580124 21005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:56.580410 21005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:56.580564 21005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:56.580790 21005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:56.581135 21005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:56.581171 21005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:56.581215 21005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:56.581245 21005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:56.587805 21005 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:45457
I20260926 06:28:56.587837 21132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:45457 every 8 connection(s)
I20260926 06:28:56.588179 21005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:28:56.593425 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.588866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:41659 (local address 127.2.39.254:40397)
0926 06:28:56.588938 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.588940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.589178 (+ 238us) server_negotiation.cc:407] Connection header received
0926 06:28:56.589728 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.589729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.589741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.589818 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:56.590452 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.591019 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.592036 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.592220 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.592288 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.593142 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:56.593143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:56.593143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:56.593147 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:56.593162 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:56.593166 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:56.593221 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.593344 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.593367 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:28:56.593528 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.588851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:40397 (local address 127.2.39.194:41659)
0926 06:28:56.589025 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.589062 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.589535 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.589870 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.589877 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.589885 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:56.590363 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.590367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.591117 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.591118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.591957 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.591959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.592296 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.592297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.592429 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.593017 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:28:56.593017 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:56.593043 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:28:56.593063 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:56.593223 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:56.593226 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:56.593228 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.593279 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.593347 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260926 06:28:56.594053 21133 heartbeater.cc:347] Connected to a master server at 127.2.39.254:40397
I20260926 06:28:56.594151 21133 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:56.594336 21133 heartbeater.cc:511] Master 127.2.39.254:40397 requested a full tablet report, sending...
I20260926 06:28:56.594717 20814 ts_manager.cc:194] Registered new tserver with Master: e3aa447cafc54bac8587cad592ab0c0e (127.2.39.194:45457)
I20260926 06:28:56.594992 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 21005
I20260926 06:28:56.595085 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:28:56.595125 20814 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.194:41659
I20260926 06:28:56.595361 2207 external_mini_cluster.cc:1428] Setting key 974e9f9ebf081f733950435e91aa2910
I20260926 06:28:56.596244 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:40397
--builtin_ntp_servers=127.2.39.212:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:28:56.675309 21136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:56.675530 21136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:56.675565 21136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:56.676698 21136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:56.676759 21136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:56.676800 21136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:28:56.678187 21136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:40397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:56.678434 21136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:56.678710 21136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:56.683974 21142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:56.683983 21144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:56.684010 21141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:56.684186 21136 server_base.cc:1034] running on GCE node
I20260926 06:28:56.684451 21136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:56.684696 21136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:56.685855 21136 hybrid_clock.cc:648] HybridClock initialized: now 1790404136685839 us; error 27 us; skew 500 ppm
I20260926 06:28:56.687072 21136 webserver.cc:460] Webserver started at http://127.2.39.195:38983/ using document root <none> and password file <none>
I20260926 06:28:56.687307 21136 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:56.687386 21136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:56.687546 21136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:56.688581 21136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "e70179410f414111a56bbadf7bb3039a"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "2fce15d5f90bfc0b4212efac630e1a98"
server_key_iv: "30a998b730d939b7fa4e7acf43e7ec76"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.688930 21136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "e70179410f414111a56bbadf7bb3039a"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "2fce15d5f90bfc0b4212efac630e1a98"
server_key_iv: "30a998b730d939b7fa4e7acf43e7ec76"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.690303 21136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:28:56.691012 21150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:56.691179 21136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:28:56.691275 21136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "e70179410f414111a56bbadf7bb3039a"
format_stamp: "Formatted at 2026-09-26 06:28:56 on dist-test-slave-fkbn"
server_key: "2fce15d5f90bfc0b4212efac630e1a98"
server_key_iv: "30a998b730d939b7fa4e7acf43e7ec76"
server_key_version: "encryptionkey@0"
I20260926 06:28:56.691355 21136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:56.714767 21136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:56.715082 21136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:56.715233 21136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:56.715461 21136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:56.715831 21136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:56.715889 21136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:56.715932 21136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:56.715960 21136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:56.721968 21136 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:34379
I20260926 06:28:56.722048 21263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:34379 every 8 connection(s)
I20260926 06:28:56.722349 21136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:28:56.727545 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.723061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:41417 (local address 127.2.39.254:40397)
0926 06:28:56.723127 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.723129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.723362 (+ 233us) server_negotiation.cc:407] Connection header received
0926 06:28:56.723882 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.723883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.723895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.723942 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:56.724624 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.725157 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.726146 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.726355 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.726432 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.727264 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:56.727265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:56.727266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:56.727270 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:56.727286 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:56.727290 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:56.727338 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.727430 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.727468 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:28:56.727747 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.723052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:40397 (local address 127.2.39.195:41417)
0926 06:28:56.723243 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.723276 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.723775 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.724020 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.724024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.724040 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:56.724525 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.724533 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.725257 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.725259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.726051 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.726053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.726437 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.726438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.726577 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.727123 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:28:56.727123 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:56.727151 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:28:56.727172 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:56.727338 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:56.727342 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:56.727345 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.727432 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.727528 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":78,"threads_started":1}
I20260926 06:28:56.728212 21264 heartbeater.cc:347] Connected to a master server at 127.2.39.254:40397
I20260926 06:28:56.728282 21264 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:56.728412 21264 heartbeater.cc:511] Master 127.2.39.254:40397 requested a full tablet report, sending...
I20260926 06:28:56.728749 20814 ts_manager.cc:194] Registered new tserver with Master: e70179410f414111a56bbadf7bb3039a (127.2.39.195:34379)
I20260926 06:28:56.729061 20814 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.195:41417
I20260926 06:28:56.731012 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 21136
I20260926 06:28:56.731094 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:28:56.731222 2207 external_mini_cluster.cc:1428] Setting key 05e43fffd321d6216838c586492430b2
I20260926 06:28:56.732093 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:28:56.736172 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.733430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.2.39.254:40397)
0926 06:28:56.733512 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.733514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.733611 (+ 97us) server_negotiation.cc:407] Connection header received
0926 06:28:56.733633 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.733633 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.733644 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.733700 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:56.733847 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.734301 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.735007 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.735188 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.735236 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.735885 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:56.735886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:56.735886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:56.735889 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:56.735899 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:56.735902 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:56.735961 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.736061 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.736073 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 06:28:56.737792 20814 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:28:56.739380 20814 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:28:56.745354 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.741711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.2.39.193:40375)
0926 06:28:56.741914 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.741919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.742263 (+ 344us) server_negotiation.cc:407] Connection header received
0926 06:28:56.742385 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.742387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.742403 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.742454 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:56.742659 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.743325 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.744341 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.744957 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.745057 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.745149 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.745230 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.745267 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":92,"threads_started":1}
I20260926 06:28:56.745343 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.741529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:40375 (local address 127.0.0.1:55568)
0926 06:28:56.742202 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.742212 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.742290 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.742540 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.742541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.742543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:56.742578 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.742579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.743416 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.743417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.744157 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.744158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.745061 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.745062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.745137 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.745150 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.745219 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.745272 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":196,"threads_started":1}
I20260926 06:28:56.745651 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.741511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:45457 (local address 127.0.0.1:47940)
0926 06:28:56.741791 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.741834 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.741896 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.742125 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.742127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.742130 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:56.742194 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.742196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.743166 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.743168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.744427 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.744429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.745382 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.745383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.745460 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.745463 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.745516 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.745555 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":1,"thread_start_us":116,"threads_started":1}
I20260926 06:28:56.745707 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.741760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.2.39.194:45457)
0926 06:28:56.741942 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.741946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.741956 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:28:56.742015 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.742021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.742034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.742119 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:56.742265 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.743075 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.744521 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.745214 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.745276 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.745350 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.745553 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.745626 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":80,"threads_started":1}
I20260926 06:28:56.746374 21067 tablet_service.cc:1467] Processing CreateTablet for tablet f1b0a7e7590e4ba4a808fc82d15bf178 (DEFAULT_TABLE table=test-table [id=f98246e5198945c08a12c7ef986651e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:56.746543 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.741591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.2.39.195:34379)
0926 06:28:56.742006 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.742009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.742018 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:28:56.742057 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.742058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.742070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.742149 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:56.743189 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.744241 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.745465 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.746210 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.746279 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.746355 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.746426 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.746463 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":72,"threads_started":1}
I20260926 06:28:56.746613 20936 tablet_service.cc:1467] Processing CreateTablet for tablet f1b0a7e7590e4ba4a808fc82d15bf178 (DEFAULT_TABLE table=test-table [id=f98246e5198945c08a12c7ef986651e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:56.746747 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.741487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:34379 (local address 127.0.0.1:36326)
0926 06:28:56.741790 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.741831 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.741897 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.742159 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.742160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.742162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:56.742206 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.742208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.744330 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.744330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.745382 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.745384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.746279 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.746281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.746341 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.746344 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.746413 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.746449 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":66,"threads_started":1}
I20260926 06:28:56.746810 21067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b0a7e7590e4ba4a808fc82d15bf178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:56.746943 20936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b0a7e7590e4ba4a808fc82d15bf178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:56.747617 21198 tablet_service.cc:1467] Processing CreateTablet for tablet f1b0a7e7590e4ba4a808fc82d15bf178 (DEFAULT_TABLE table=test-table [id=f98246e5198945c08a12c7ef986651e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:56.747925 21198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1b0a7e7590e4ba4a808fc82d15bf178. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:56.749377 21278 tablet_bootstrap.cc:492] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511: Bootstrap starting.
I20260926 06:28:56.749958 21278 tablet_bootstrap.cc:654] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:56.750188 21279 tablet_bootstrap.cc:492] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: Bootstrap starting.
I20260926 06:28:56.750269 21278 log.cc:824] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:56.750806 21279 tablet_bootstrap.cc:654] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:56.751015 21278 tablet_bootstrap.cc:492] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511: No bootstrap required, opened a new log
I20260926 06:28:56.751088 21278 ts_tablet_manager.cc:1397] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 06:28:56.751112 21279 log.cc:824] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:56.751327 21281 tablet_bootstrap.cc:492] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: Bootstrap starting.
I20260926 06:28:56.751881 21279 tablet_bootstrap.cc:492] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: No bootstrap required, opened a new log
I20260926 06:28:56.751956 21279 ts_tablet_manager.cc:1397] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:28:56.752090 21281 tablet_bootstrap.cc:654] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:56.752471 21281 log.cc:824] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:56.753301 21281 tablet_bootstrap.cc:492] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: No bootstrap required, opened a new log
I20260926 06:28:56.753180 21278 raft_consensus.cc:348] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } }
I20260926 06:28:56.753345 21278 raft_consensus.cc:374] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:56.753396 21278 raft_consensus.cc:729] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7365e88f7feb4f3fa28855b2478f7511, State: Initialized, Role: FOLLOWER
I20260926 06:28:56.753504 21281 ts_tablet_manager.cc:1397] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:28:56.753512 21278 consensus_queue.cc:260] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } }
I20260926 06:28:56.753784 21278 ts_tablet_manager.cc:1428] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 06:28:56.753935 21002 heartbeater.cc:503] Master 127.2.39.254:40397 was elected leader, sending a full tablet report...
I20260926 06:28:56.754671 21279 raft_consensus.cc:348] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } }
I20260926 06:28:56.754827 21279 raft_consensus.cc:374] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:56.754869 21279 raft_consensus.cc:729] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3aa447cafc54bac8587cad592ab0c0e, State: Initialized, Role: FOLLOWER
I20260926 06:28:56.754981 21279 consensus_queue.cc:260] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } }
I20260926 06:28:56.755234 21279 ts_tablet_manager.cc:1428] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 06:28:56.755278 21133 heartbeater.cc:503] Master 127.2.39.254:40397 was elected leader, sending a full tablet report...
I20260926 06:28:56.756093 21281 raft_consensus.cc:348] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } }
I20260926 06:28:56.756258 21281 raft_consensus.cc:374] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:56.756282 21281 raft_consensus.cc:729] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70179410f414111a56bbadf7bb3039a, State: Initialized, Role: FOLLOWER
I20260926 06:28:56.756398 21281 consensus_queue.cc:260] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } }
I20260926 06:28:56.756628 21281 ts_tablet_manager.cc:1428] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:28:56.756744 21264 heartbeater.cc:503] Master 127.2.39.254:40397 was elected leader, sending a full tablet report...
I20260926 06:28:56.834847 21284 raft_consensus.cc:482] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:56.834964 21284 raft_consensus.cc:504] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } }
I20260926 06:28:56.835382 21284 leader_election.cc:288] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e70179410f414111a56bbadf7bb3039a (127.2.39.195:34379), e3aa447cafc54bac8587cad592ab0c0e (127.2.39.194:45457)
W20260926 06:28:56.838724 21134 tablet.cc:2369] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:28:56.839236 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.835624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:46573 (local address 127.2.39.194:45457)
0926 06:28:56.835669 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.835671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.835819 (+ 148us) server_negotiation.cc:407] Connection header received
0926 06:28:56.835861 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.835862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.835871 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.835917 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:56.836081 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.836780 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.838326 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.838897 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.838957 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.838964 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.839126 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.839169 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:28:56.839236 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.835657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:50537 (local address 127.2.39.195:34379)
0926 06:28:56.835703 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.835705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.835726 (+ 21us) server_negotiation.cc:407] Connection header received
0926 06:28:56.835758 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.835759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.835768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.835818 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:56.836003 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.836781 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.838326 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.838873 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.838951 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.838959 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.839126 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.839169 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:28:56.839236 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.835559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:34379 (local address 127.2.39.193:50537)
0926 06:28:56.835618 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.835633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.835681 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.835870 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.835871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.835873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:56.835947 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.835949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.836864 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.836864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.838233 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.838236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.838952 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.838953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.839035 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.839039 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.839103 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.839145 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":75}
I20260926 06:28:56.839242 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.835575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:45457 (local address 127.2.39.193:46573)
0926 06:28:56.835733 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.835744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.835810 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.835934 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.835935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.835937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:56.836009 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.836011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.836864 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.836864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.838237 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.838239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.838991 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.838992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.839057 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.839060 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.839113 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.839143 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":67,"thread_start_us":96,"threads_started":1}
I20260926 06:28:56.839653 21218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b0a7e7590e4ba4a808fc82d15bf178" candidate_uuid: "7365e88f7feb4f3fa28855b2478f7511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70179410f414111a56bbadf7bb3039a" is_pre_election: true
I20260926 06:28:56.839780 21286 raft_consensus.cc:482] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:56.839789 21087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b0a7e7590e4ba4a808fc82d15bf178" candidate_uuid: "7365e88f7feb4f3fa28855b2478f7511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3aa447cafc54bac8587cad592ab0c0e" is_pre_election: true
I20260926 06:28:56.839869 21286 raft_consensus.cc:504] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } }
I20260926 06:28:56.839916 21087 raft_consensus.cc:2454] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365e88f7feb4f3fa28855b2478f7511 in term 0.
I20260926 06:28:56.839973 21218 raft_consensus.cc:2454] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7365e88f7feb4f3fa28855b2478f7511 in term 0.
I20260926 06:28:56.840171 20889 leader_election.cc:302] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365e88f7feb4f3fa28855b2478f7511, e3aa447cafc54bac8587cad592ab0c0e; no voters:
I20260926 06:28:56.840286 21284 raft_consensus.cc:2781] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:28:56.840340 21284 raft_consensus.cc:482] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:28:56.840368 21284 raft_consensus.cc:3037] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:56.840452 21286 leader_election.cc:288] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3aa447cafc54bac8587cad592ab0c0e (127.2.39.194:45457), 7365e88f7feb4f3fa28855b2478f7511 (127.2.39.193:40375)
I20260926 06:28:56.841320 21284 raft_consensus.cc:504] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } }
I20260926 06:28:56.841466 21284 leader_election.cc:288] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [CANDIDATE]: Term 1 election: Requested vote from peers e70179410f414111a56bbadf7bb3039a (127.2.39.195:34379), e3aa447cafc54bac8587cad592ab0c0e (127.2.39.194:45457)
I20260926 06:28:56.841671 21087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b0a7e7590e4ba4a808fc82d15bf178" candidate_uuid: "7365e88f7feb4f3fa28855b2478f7511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3aa447cafc54bac8587cad592ab0c0e"
I20260926 06:28:56.841694 21218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b0a7e7590e4ba4a808fc82d15bf178" candidate_uuid: "7365e88f7feb4f3fa28855b2478f7511" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70179410f414111a56bbadf7bb3039a"
I20260926 06:28:56.841750 21218 raft_consensus.cc:3037] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:56.841753 21087 raft_consensus.cc:3037] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:56.842614 21087 raft_consensus.cc:2454] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365e88f7feb4f3fa28855b2478f7511 in term 1.
I20260926 06:28:56.842614 21218 raft_consensus.cc:2454] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7365e88f7feb4f3fa28855b2478f7511 in term 1.
I20260926 06:28:56.842803 20892 leader_election.cc:302] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7365e88f7feb4f3fa28855b2478f7511, e70179410f414111a56bbadf7bb3039a; no voters:
I20260926 06:28:56.842890 21284 raft_consensus.cc:2781] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:56.843014 21284 raft_consensus.cc:686] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 1 LEADER]: Becoming Leader. State: Replica: 7365e88f7feb4f3fa28855b2478f7511, State: Running, Role: LEADER
I20260926 06:28:56.843108 21284 consensus_queue.cc:237] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } }
I20260926 06:28:56.843604 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.840282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:45457 (local address 127.2.39.195:54765)
0926 06:28:56.840318 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.840329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.840368 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.840589 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.840590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.840591 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:56.840644 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.840645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.841442 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.841444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.842632 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.842635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.843345 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.843346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.843411 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.843414 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.843498 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.843538 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 06:28:56.843621 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.840387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:54765 (local address 127.2.39.194:45457)
0926 06:28:56.840450 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.840452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.840460 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:28:56.840480 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.840482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.840491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.840545 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:56.840714 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.841350 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.842726 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.843251 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.843311 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.843319 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.843524 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.843567 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:28:56.843921 21087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b0a7e7590e4ba4a808fc82d15bf178" candidate_uuid: "e70179410f414111a56bbadf7bb3039a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3aa447cafc54bac8587cad592ab0c0e" is_pre_election: true
I20260926 06:28:56.844014 21087 raft_consensus.cc:2379] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e70179410f414111a56bbadf7bb3039a in current term 1: Already voted for candidate 7365e88f7feb4f3fa28855b2478f7511 in this term.
I20260926 06:28:56.843977 20814 catalog_manager.cc:5495] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7365e88f7feb4f3fa28855b2478f7511 (127.2.39.193). New cstate: current_term: 1 leader_uuid: "7365e88f7feb4f3fa28855b2478f7511" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7365e88f7feb4f3fa28855b2478f7511" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 40375 } health_report { overall_health: HEALTHY } } }
I20260926 06:28:56.844170 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.840549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:40375 (local address 127.2.39.195:43845)
0926 06:28:56.840692 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.840703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:56.840749 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:56.840994 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:56.840995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:56.840997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:56.841071 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.841073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.842064 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.842065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.843211 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:56.843213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.843952 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.843953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:56.844018 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.844021 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:56.844070 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 06:28:56.844102 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260926 06:28:56.844398 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.840744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:43845 (local address 127.2.39.193:40375)
0926 06:28:56.840874 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.840875 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.840882 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:28:56.840901 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.840902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.840910 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.840990 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:56.841150 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.841970 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.843303 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.843841 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.843903 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.843909 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.844301 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.844342 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 06:28:56.844763 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1b0a7e7590e4ba4a808fc82d15bf178" candidate_uuid: "e70179410f414111a56bbadf7bb3039a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7365e88f7feb4f3fa28855b2478f7511" is_pre_election: true
I20260926 06:28:56.845031 21152 leader_election.cc:302] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e70179410f414111a56bbadf7bb3039a; no voters: 7365e88f7feb4f3fa28855b2478f7511, e3aa447cafc54bac8587cad592ab0c0e
I20260926 06:28:56.845170 21286 raft_consensus.cc:2726] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 06:28:56.863214 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.860831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.2.39.193:40375)
0926 06:28:56.860876 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:56.860878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:56.860888 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:28:56.860973 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:56.860974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:56.860990 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:56.861064 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:28:56.861197 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.861713 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.862438 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:56.862642 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:56.862715 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:56.862844 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:28:56.862947 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:28:56.863057 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:56.863110 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 06:28:56.863122 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:28:56.865043 21087 raft_consensus.cc:1261] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [term 1 FOLLOWER]: Refusing update from remote peer 7365e88f7feb4f3fa28855b2478f7511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:56.865064 21218 raft_consensus.cc:1261] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 1 FOLLOWER]: Refusing update from remote peer 7365e88f7feb4f3fa28855b2478f7511: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:56.865379 21284 consensus_queue.cc:1035] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3aa447cafc54bac8587cad592ab0c0e" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 45457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:56.865546 21289 consensus_queue.cc:1035] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e70179410f414111a56bbadf7bb3039a" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 34379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:56.867883 21293 mvcc.cc:205] Tried to move back new op lower bound from 7333495344596078592 to 7333495344509837312. Current Snapshot: MvccSnapshot[applied={T|T < 7333495344596078592}]
I20260926 06:28:56.870323 20814 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:56.870428 20814 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:56.871142 20814 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53c6853d997c4fafb20984119839987c: Sending DeleteTablet for 3 replicas of tablet f1b0a7e7590e4ba4a808fc82d15bf178
I20260926 06:28:56.871448 21067 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b0a7e7590e4ba4a808fc82d15bf178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:56 UTC) from {username='slave'} at 127.0.0.1:47940
I20260926 06:28:56.871445 20936 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b0a7e7590e4ba4a808fc82d15bf178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:56 UTC) from {username='slave'} at 127.0.0.1:55568
I20260926 06:28:56.871673 21303 tablet_replica.cc:330] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511: stopping tablet replica
I20260926 06:28:56.871677 21198 tablet_service.cc:1514] Processing DeleteTablet for tablet f1b0a7e7590e4ba4a808fc82d15bf178 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:56 UTC) from {username='slave'} at 127.0.0.1:36326
I20260926 06:28:56.871831 21303 raft_consensus.cc:2229] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:28:56.871881 21304 tablet_replica.cc:330] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: stopping tablet replica
I20260926 06:28:56.871986 21304 raft_consensus.cc:2229] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:56.872009 21303 raft_consensus.cc:2258] T f1b0a7e7590e4ba4a808fc82d15bf178 P 7365e88f7feb4f3fa28855b2478f7511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:56.872090 21304 raft_consensus.cc:2258] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:56.872224 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 20874
I20260926 06:28:56.873001 21302 tablet_replica.cc:330] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: stopping tablet replica
I20260926 06:28:56.873059 21304 ts_tablet_manager.cc:1905] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:56.873090 21302 raft_consensus.cc:2229] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:56.873180 21302 raft_consensus.cc:2258] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:56.873533 21302 ts_tablet_manager.cc:1905] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:56.875177 21304 ts_tablet_manager.cc:1918] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:56.875237 21304 log.cc:1196] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/f1b0a7e7590e4ba4a808fc82d15bf178
I20260926 06:28:56.875401 21302 ts_tablet_manager.cc:1918] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:56.875474 21302 log.cc:1196] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/f1b0a7e7590e4ba4a808fc82d15bf178
I20260926 06:28:56.875526 21304 ts_tablet_manager.cc:1939] T f1b0a7e7590e4ba4a808fc82d15bf178 P e70179410f414111a56bbadf7bb3039a: Deleting consensus metadata
I20260926 06:28:56.875734 21302 ts_tablet_manager.cc:1939] T f1b0a7e7590e4ba4a808fc82d15bf178 P e3aa447cafc54bac8587cad592ab0c0e: Deleting consensus metadata
I20260926 06:28:56.875891 20801 catalog_manager.cc:4841] TS e70179410f414111a56bbadf7bb3039a (127.2.39.195:34379): tablet f1b0a7e7590e4ba4a808fc82d15bf178 (table test-table [id=f98246e5198945c08a12c7ef986651e6]) successfully deleted
I20260926 06:28:56.876228 20798 catalog_manager.cc:4841] TS e3aa447cafc54bac8587cad592ab0c0e (127.2.39.194:45457): tablet f1b0a7e7590e4ba4a808fc82d15bf178 (table test-table [id=f98246e5198945c08a12c7ef986651e6]) successfully deleted
W20260926 06:28:56.878463 20799 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:40375 (error 108)
I20260926 06:28:56.878589 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 21005
I20260926 06:28:56.879647 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:56.879508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:40375 (local address 127.0.0.1:55582)
0926 06:28:56.879569 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:56.879614 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:40375: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 06:28:56.879801 20799 catalog_manager.cc:4584] TS 7365e88f7feb4f3fa28855b2478f7511 (127.2.39.193:40375): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1b0a7e7590e4ba4a808fc82d15bf178: Network error: Client connection negotiation failed: client connection to 127.2.39.193:40375: connect: Connection refused (error 111)
I20260926 06:28:56.884095 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 21136
I20260926 06:28:56.888442 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 20783
2026-09-26T06:28:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (729 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:28:56 dist-test-slave-fkbn krb5kdc[21308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:28:56 dist-test-slave-fkbn krb5kdc[21308](info): set up 2 sockets
Sep 26 06:28:56 dist-test-slave-fkbn krb5kdc[21308](info): commencing operation
krb5kdc: starting...
W20260926 06:28:57.933706 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:28:58 dist-test-slave-fkbn krb5kdc[21308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:28:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:28:58.052356 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:37629
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:37629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:58.133041 21321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:58.133234 21321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:58.133255 21321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:58.134420 21321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:28:58.134461 21321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:58.134475 21321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:58.134488 21321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:28:58.134500 21321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:28:58.136077 21321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:37629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:37629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:58.136353 21321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:58.136581 21321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:58.141414 21329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:58.141471 21326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:58.141418 21327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:58.141649 21321 server_base.cc:1034] running on GCE node
I20260926 06:28:58.141896 21321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:58.142175 21321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:58.143313 21321 hybrid_clock.cc:648] HybridClock initialized: now 1790404138143293 us; error 37 us; skew 500 ppm
Sep 26 06:28:58 dist-test-slave-fkbn krb5kdc[21308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404138, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:58.145655 21321 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:28:58.146075 21321 webserver.cc:460] Webserver started at http://127.2.39.254:44493/ using document root <none> and password file <none>
I20260926 06:28:58.146292 21321 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:58.146359 21321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:58.146522 21321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:58.147526 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "206bf5b4d82244e8bc1f8b476c11a583"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "ec1e9925e43c4efebf0f48ece63e79ac"
server_key_iv: "c5850c9960c7e26033475c1a147516f8"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.147883 21321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "206bf5b4d82244e8bc1f8b476c11a583"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "ec1e9925e43c4efebf0f48ece63e79ac"
server_key_iv: "c5850c9960c7e26033475c1a147516f8"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.149164 21321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:28:58.149899 21336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.150077 21321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.150174 21321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "206bf5b4d82244e8bc1f8b476c11a583"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "ec1e9925e43c4efebf0f48ece63e79ac"
server_key_iv: "c5850c9960c7e26033475c1a147516f8"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.150251 21321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:58.160899 21321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:58.162647 21321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:58.162806 21321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:58.166658 21321 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:37629
I20260926 06:28:58.166706 21388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:37629 every 8 connection(s)
I20260926 06:28:58.167021 21321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:28:58.167757 21389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:58.170033 21389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583: Bootstrap starting.
I20260926 06:28:58.170590 21389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:58.170876 21389 log.cc:824] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:58.171558 21389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583: No bootstrap required, opened a new log
I20260926 06:28:58.173209 21389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206bf5b4d82244e8bc1f8b476c11a583" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37629 } }
I20260926 06:28:58.173345 21389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:58.173367 21389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206bf5b4d82244e8bc1f8b476c11a583, State: Initialized, Role: FOLLOWER
I20260926 06:28:58.173545 21389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206bf5b4d82244e8bc1f8b476c11a583" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37629 } }
I20260926 06:28:58.173652 21389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:28:58.173696 21389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:28:58.173745 21389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:58.174263 21389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206bf5b4d82244e8bc1f8b476c11a583" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37629 } }
I20260926 06:28:58.174371 21389 leader_election.cc:302] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 206bf5b4d82244e8bc1f8b476c11a583; no voters:
I20260926 06:28:58.174552 21389 leader_election.cc:288] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:28:58.174619 21392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:58.174767 21392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [term 1 LEADER]: Becoming Leader. State: Replica: 206bf5b4d82244e8bc1f8b476c11a583, State: Running, Role: LEADER
I20260926 06:28:58.174801 21389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:28:58.174854 21392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206bf5b4d82244e8bc1f8b476c11a583" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37629 } }
I20260926 06:28:58.175263 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 206bf5b4d82244e8bc1f8b476c11a583. Latest consensus state: current_term: 1 leader_uuid: "206bf5b4d82244e8bc1f8b476c11a583" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206bf5b4d82244e8bc1f8b476c11a583" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37629 } } }
I20260926 06:28:58.175351 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:58.175493 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "206bf5b4d82244e8bc1f8b476c11a583" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "206bf5b4d82244e8bc1f8b476c11a583" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37629 } } }
I20260926 06:28:58.175585 21401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:28:58.175609 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:58.176103 21401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:28:58.176846 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 21321
I20260926 06:28:58.176977 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:28:58.177173 2207 external_mini_cluster.cc:1428] Setting key c634b30fce1664d4952562c6cc145386
I20260926 06:28:58.177928 21401 catalog_manager.cc:1347] Generated new cluster ID: 95edf100774a4c5ba584b77815a93ea3
I20260926 06:28:58.177980 21401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:28:58 dist-test-slave-fkbn krb5kdc[21308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:58.189512 21401 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:28:58.190208 21401 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:28:58.191269 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.177784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53442 (local address 127.2.39.254:37629)
0926 06:28:58.177941 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.177949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.181837 (+ 3888us) server_negotiation.cc:407] Connection header received
0926 06:28:58.181991 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.181997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.182048 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.182187 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:58.182844 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.184065 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.184813 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.185254 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.185377 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.187741 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:58.187745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:58.187745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:58.187765 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:58.189715 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:58.190087 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:58.190090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:58.190090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:58.190186 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:58.190455 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:58.190456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:58.190457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:58.190672 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:58.190841 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.190938 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.191089 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260926 06:28:58.201928 21401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583: Generated new TSK 0
I20260926 06:28:58.202118 21401 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.39.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
Entry for principal kudu/127.2.39.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab.
I20260926 06:28:58.228556 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:37629
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:58.308416 21415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:58.308645 21415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:58.308684 21415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:58.309876 21415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:58.309942 21415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:58.309994 21415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:28:58.311432 21415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.193.keytab
--principal=kudu/127.2.39.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:37629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:58.311691 21415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:58.311913 21415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:58.316983 21420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:58.317035 21421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:58.317051 21415 server_base.cc:1034] running on GCE node
W20260926 06:28:58.317013 21423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:58.317373 21415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:58.317651 21415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:58.318800 21415 hybrid_clock.cc:648] HybridClock initialized: now 1790404138318787 us; error 28 us; skew 500 ppm
Sep 26 06:28:58 dist-test-slave-fkbn krb5kdc[21308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404138, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:58.321167 21415 init.cc:377] Logged in from keytab as kudu/127.2.39.193@KRBTEST.COM (short username kudu)
I20260926 06:28:58.321700 21415 webserver.cc:460] Webserver started at http://127.2.39.193:40627/ using document root <none> and password file <none>
I20260926 06:28:58.321926 21415 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:58.322000 21415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:58.322132 21415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:58.323151 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "f43c6507bf464379a91e46dcd15f770c"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "646911c9aec56f816b45b295b671d408"
server_key_iv: "2a27efec7e470b6eb80756152a7e6778"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.323510 21415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "f43c6507bf464379a91e46dcd15f770c"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "646911c9aec56f816b45b295b671d408"
server_key_iv: "2a27efec7e470b6eb80756152a7e6778"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.324755 21415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:28:58.325469 21430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.325660 21415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.325755 21415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "f43c6507bf464379a91e46dcd15f770c"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "646911c9aec56f816b45b295b671d408"
server_key_iv: "2a27efec7e470b6eb80756152a7e6778"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.325839 21415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:58.351864 21415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:58.353914 21415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:58.354081 21415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:58.354322 21415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:58.354660 21415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:58.354715 21415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.354756 21415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:58.354785 21415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.360422 21415 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:45811
I20260926 06:28:58.360478 21543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:45811 every 8 connection(s)
I20260926 06:28:58.360773 21415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:28:58.363340 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 21415
I20260926 06:28:58.363478 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:28:58.363655 2207 external_mini_cluster.cc:1428] Setting key 4e433be384ef45ab416f98bf9c5bfe22
Sep 26 06:28:58 dist-test-slave-fkbn krb5kdc[21308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404138, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.193@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:58.371137 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.361481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:59587 (local address 127.2.39.254:37629)
0926 06:28:58.361567 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.361569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.361764 (+ 195us) server_negotiation.cc:407] Connection header received
0926 06:28:58.362287 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.362289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.362301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.362355 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:58.363333 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.364053 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.365463 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.365761 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.365848 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.368098 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:58.368100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:58.368100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:58.368108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:58.369786 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:58.370168 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:58.370170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:58.370171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:58.370234 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:58.370466 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:58.370467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:58.370467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:58.370614 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:58.370687 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.370918 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.371026 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 06:28:58.371157 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.361437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:37629 (local address 127.2.39.193:59587)
0926 06:28:58.361630 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:58.361681 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:58.362180 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:58.362425 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:58.362429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:58.363025 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:58.363225 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.363230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.364144 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.364145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.365378 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.365381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.365853 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.365855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.365998 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.366508 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:58.366516 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:58.368024 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:58.369864 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:58.369869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:58.369871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:58.370095 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:58.370303 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:58.370303 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:58.370304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:58.370397 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:58.370680 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:58.370684 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:58.370791 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 06:28:58.370871 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 06:28:58.371004 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":91,"threads_started":1}
I20260926 06:28:58.371739 21544 heartbeater.cc:347] Connected to a master server at 127.2.39.254:37629
I20260926 06:28:58.371848 21544 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:58.372041 21544 heartbeater.cc:511] Master 127.2.39.254:37629 requested a full tablet report, sending...
I20260926 06:28:58.372584 21353 ts_manager.cc:194] Registered new tserver with Master: f43c6507bf464379a91e46dcd15f770c (127.2.39.193:45811)
WARNING: no policy specified for kudu/127.2.39.194@KRBTEST.COM; defaulting to no policy
I20260926 06:28:58.373287 21353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.193@KRBTEST.COM'} at 127.2.39.193:59587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
Entry for principal kudu/127.2.39.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab.
I20260926 06:28:58.388615 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:37629
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:58.468894 21549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:58.469080 21549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:58.469102 21549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:58.470255 21549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:58.470295 21549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:58.470321 21549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:28:58.471657 21549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.194.keytab
--principal=kudu/127.2.39.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:37629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:58.471832 21549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:58.471999 21549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:58.477348 21554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:58.477335 21555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:58.477348 21557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:58.477407 21549 server_base.cc:1034] running on GCE node
I20260926 06:28:58.477813 21549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:58.478073 21549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:58.479236 21549 hybrid_clock.cc:648] HybridClock initialized: now 1790404138479221 us; error 27 us; skew 500 ppm
Sep 26 06:28:58 dist-test-slave-fkbn krb5kdc[21308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404138, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:58.481813 21549 init.cc:377] Logged in from keytab as kudu/127.2.39.194@KRBTEST.COM (short username kudu)
I20260926 06:28:58.482195 21549 webserver.cc:460] Webserver started at http://127.2.39.194:33571/ using document root <none> and password file <none>
I20260926 06:28:58.482381 21549 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:58.482420 21549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:58.482559 21549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:58.483572 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "64d6fdf3632646deb45f31c945847f28"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "fa79a3f36ee95bbedf409c150c0a9c43"
server_key_iv: "d28b23da16698e152e5db94dcd1d1e97"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.483916 21549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "64d6fdf3632646deb45f31c945847f28"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "fa79a3f36ee95bbedf409c150c0a9c43"
server_key_iv: "d28b23da16698e152e5db94dcd1d1e97"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.485167 21549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:28:58.485906 21564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.486094 21549 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.486176 21549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "64d6fdf3632646deb45f31c945847f28"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "fa79a3f36ee95bbedf409c150c0a9c43"
server_key_iv: "d28b23da16698e152e5db94dcd1d1e97"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.486227 21549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:58.499104 21549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:58.500765 21549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:58.500865 21549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:58.501030 21549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:58.501278 21549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:58.501307 21549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.501327 21549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:58.501339 21549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.506872 21549 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:42315
I20260926 06:28:58.506949 21677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:42315 every 8 connection(s)
I20260926 06:28:58.507244 21549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:28:58.513072 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 21549
I20260926 06:28:58.513204 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:28:58.513402 2207 external_mini_cluster.cc:1428] Setting key d05389d944c37194f56ab63f2620b669
Sep 26 06:28:58 dist-test-slave-fkbn krb5kdc[21308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404138, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.194@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:58.518034 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.507988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:54969 (local address 127.2.39.254:37629)
0926 06:28:58.508041 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.508043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.508256 (+ 213us) server_negotiation.cc:407] Connection header received
0926 06:28:58.508864 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.508866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.508880 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.508954 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:58.510196 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.510982 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.512192 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.512384 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.512443 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.514972 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:58.514974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:58.514974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:58.514983 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:58.516613 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:58.516982 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:58.516983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:58.516983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:58.517054 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:58.517277 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:58.517278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:58.517278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:58.517425 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:58.517497 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.517823 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.517915 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:28:58.518054 21680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.507966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:37629 (local address 127.2.39.194:54969)
0926 06:28:58.508140 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:58.508173 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:58.508709 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:58.509032 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:58.509036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:58.509826 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:58.510091 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.510096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.511065 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.511066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.512109 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.512112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.512466 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.512467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.512564 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.512997 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:58.513007 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:58.514876 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:58.516693 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:58.516698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:58.516700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:58.516906 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:58.517121 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:58.517121 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:58.517121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:58.517212 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:58.517491 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:58.517494 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:58.517671 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 06:28:58.517772 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 06:28:58.517889 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for kudu/127.2.39.195@KRBTEST.COM; defaulting to no policy
I20260926 06:28:58.521363 21678 heartbeater.cc:347] Connected to a master server at 127.2.39.254:37629
I20260926 06:28:58.521466 21678 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:58.521667 21678 heartbeater.cc:511] Master 127.2.39.254:37629 requested a full tablet report, sending...
I20260926 06:28:58.522081 21353 ts_manager.cc:194] Registered new tserver with Master: 64d6fdf3632646deb45f31c945847f28 (127.2.39.194:42315)
I20260926 06:28:58.522459 21353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.194@KRBTEST.COM'} at 127.2.39.194:54969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
Entry for principal kudu/127.2.39.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab.
I20260926 06:28:58.538398 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:37629
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:28:58.618160 21683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:58.618350 21683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:58.618371 21683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:58.619450 21683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:58.619491 21683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:58.619516 21683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:28:58.620863 21683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.195.keytab
--principal=kudu/127.2.39.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:37629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:58.621040 21683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:58.621206 21683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:58.625881 21688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:58.625887 21691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:58.625890 21689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:58.626215 21683 server_base.cc:1034] running on GCE node
I20260926 06:28:58.626398 21683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:58.626647 21683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:58.627810 21683 hybrid_clock.cc:648] HybridClock initialized: now 1790404138627791 us; error 26 us; skew 500 ppm
Sep 26 06:28:58 dist-test-slave-fkbn krb5kdc[21308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404138, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:28:58.630250 21683 init.cc:377] Logged in from keytab as kudu/127.2.39.195@KRBTEST.COM (short username kudu)
I20260926 06:28:58.630635 21683 webserver.cc:460] Webserver started at http://127.2.39.195:45337/ using document root <none> and password file <none>
I20260926 06:28:58.630869 21683 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:58.630914 21683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:58.631089 21683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:58.632078 21683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "666c7f486a9c4bdaaee9ea119edcd538"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "a3046a1a1ebcd0f1dba91d8d9a7e38ae"
server_key_iv: "fddd15999e9852d10a8944b788fbbeb8"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.632416 21683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "666c7f486a9c4bdaaee9ea119edcd538"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "a3046a1a1ebcd0f1dba91d8d9a7e38ae"
server_key_iv: "fddd15999e9852d10a8944b788fbbeb8"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.633703 21683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:28:58.634573 21698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.634750 21683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:28:58.634837 21683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "666c7f486a9c4bdaaee9ea119edcd538"
format_stamp: "Formatted at 2026-09-26 06:28:58 on dist-test-slave-fkbn"
server_key: "a3046a1a1ebcd0f1dba91d8d9a7e38ae"
server_key_iv: "fddd15999e9852d10a8944b788fbbeb8"
server_key_version: "encryptionkey@0"
I20260926 06:28:58.634909 21683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:58.646245 21683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:58.647889 21683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:58.648043 21683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:58.648241 21683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:58.648535 21683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:58.648592 21683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.648633 21683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:58.648660 21683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:58.654732 21683 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:39957
I20260926 06:28:58.654774 21811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:39957 every 8 connection(s)
I20260926 06:28:58.655090 21683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
Sep 26 06:28:58 dist-test-slave-fkbn krb5kdc[21308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404138, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.195@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:28:58.662611 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 21683
I20260926 06:28:58.662726 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:28:58.662930 2207 external_mini_cluster.cc:1428] Setting key 892e40303496fadbf18337a7b0541284
I20260926 06:28:58.664250 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.655695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:33291 (local address 127.2.39.254:37629)
0926 06:28:58.655774 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.655777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.655953 (+ 176us) server_negotiation.cc:407] Connection header received
0926 06:28:58.656395 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.656395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.656405 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.656463 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:58.657376 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.657922 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.658864 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.659051 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.659101 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.661013 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:58.661015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:58.661015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:58.661021 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:58.662709 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:58.663229 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:58.663230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:58.663230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:58.663287 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:58.663611 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:58.663612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:58.663612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:58.663750 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:58.663820 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.664128 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.664190 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:28:58.664461 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.655677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:37629 (local address 127.2.39.195:33291)
0926 06:28:58.655832 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:58.655866 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:58.656315 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:58.656519 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:58.656522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:58.657106 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:58.657283 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.657288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.658015 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.658015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.658793 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.658795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.659119 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.659120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.659213 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.659638 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:28:58.659645 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:58.660931 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:58.662822 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:58.662829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:58.662831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:58.663140 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:58.663368 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:28:58.663369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:28:58.663369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:28:58.663534 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:28:58.663818 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:58.663823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:58.663980 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 06:28:58.664095 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 06:28:58.664217 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":98,"threads_started":1}
I20260926 06:28:58.664925 21812 heartbeater.cc:347] Connected to a master server at 127.2.39.254:37629
I20260926 06:28:58.665024 21812 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:58.665163 21812 heartbeater.cc:511] Master 127.2.39.254:37629 requested a full tablet report, sending...
I20260926 06:28:58.665508 21353 ts_manager.cc:194] Registered new tserver with Master: 666c7f486a9c4bdaaee9ea119edcd538 (127.2.39.195:39957)
I20260926 06:28:58.665910 21353 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.39.195@KRBTEST.COM'} at 127.2.39.195:33291
I20260926 06:28:58.675293 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:28:58.685127 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.677637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.2.39.254:37629)
0926 06:28:58.677686 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.677688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.677821 (+ 133us) server_negotiation.cc:407] Connection header received
0926 06:28:58.677897 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.677898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.677909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.677954 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:58.678735 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.679673 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.680438 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.680650 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.680736 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.682115 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:58.682116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:58.682117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:28:58.682124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:58.683807 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:58.684250 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:58.684251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:58.684251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:58.684311 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:28:58.684535 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:28:58.684536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:28:58.684537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:28:58.684696 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:58.684806 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.684936 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.685021 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:28:58.694028 21353 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:28:58.696209 21353 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:28:58.703367 21820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.698590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:42315 (local address 127.0.0.1:59516)
0926 06:28:58.698711 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:58.698757 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:58.698829 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:58.699198 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:58.699201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:58.699203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:58.699285 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.699287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.700583 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.700584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.702133 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.702136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.703068 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.703069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.703164 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.703180 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 06:28:58.703228 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:28:58.703278 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":92,"spinlock_wait_cycles":384,"thread_start_us":59,"threads_started":1}
I20260926 06:28:58.703702 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.698843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.2.39.194:42315)
0926 06:28:58.699024 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.699030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.699040 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:28:58.699077 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.699079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.699093 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.699193 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:58.699375 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.700432 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.702237 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.702967 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.703491 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.703557 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.703586 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.703628 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260926 06:28:58.704018 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.698683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.2.39.193:45811)
0926 06:28:58.699294 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.699300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.699310 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:28:58.699354 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.699356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.699373 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.699447 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:58.699605 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.700744 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.702357 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.703101 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.703154 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.703216 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.703728 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.703939 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":82,"threads_started":1}
I20260926 06:28:58.704217 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.698590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45811 (local address 127.0.0.1:49346)
0926 06:28:58.698940 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:58.698949 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:58.698988 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:58.699445 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:58.699445 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:58.699447 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:58.699514 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.699516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.700864 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.700865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.702255 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.702258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.703273 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.703274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.703342 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.703346 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:58.704107 (+ 761us) client_negotiation.cc:240] Negotiation successful
0926 06:28:58.704149 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":37,"thread_start_us":58,"threads_started":1}
I20260926 06:28:58.704416 21612 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae6cdef38584f74a44dcacb86a211c0 (DEFAULT_TABLE table=test-table [id=f77d85018bba473188a2e3187ef954d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:58.704784 21612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae6cdef38584f74a44dcacb86a211c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:58.704912 21478 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae6cdef38584f74a44dcacb86a211c0 (DEFAULT_TABLE table=test-table [id=f77d85018bba473188a2e3187ef954d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:58.705259 21478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae6cdef38584f74a44dcacb86a211c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:58.705746 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.698789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:39957 (local address 127.0.0.1:55852)
0926 06:28:58.699099 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:58.699111 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:58.699164 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:58.699816 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:58.699817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:58.699818 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:58.699855 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.699857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.701102 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.701103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.702361 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.702362 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.703595 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.703597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.703652 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.703655 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:58.703697 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 06:28:58.703735 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":35,"threads_started":1}
I20260926 06:28:58.706039 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.698947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.2.39.195:39957)
0926 06:28:58.699596 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.699600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.699610 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:28:58.699650 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.699652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.699668 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.699736 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:58.699933 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.700990 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.702444 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.703118 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.703177 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.703222 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.705916 (+ 2694us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.705960 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":74,"threads_started":1}
I20260926 06:28:58.706632 21746 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae6cdef38584f74a44dcacb86a211c0 (DEFAULT_TABLE table=test-table [id=f77d85018bba473188a2e3187ef954d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:58.706933 21746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae6cdef38584f74a44dcacb86a211c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:58.708642 21826 tablet_bootstrap.cc:492] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: Bootstrap starting.
I20260926 06:28:58.709295 21826 tablet_bootstrap.cc:654] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:58.709918 21826 log.cc:824] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:58.710137 21828 tablet_bootstrap.cc:492] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: Bootstrap starting.
I20260926 06:28:58.710698 21826 tablet_bootstrap.cc:492] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: No bootstrap required, opened a new log
I20260926 06:28:58.710759 21826 ts_tablet_manager.cc:1397] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 06:28:58.710803 21828 tablet_bootstrap.cc:654] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:58.711373 21828 log.cc:824] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:58.712646 21826 raft_consensus.cc:348] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 } } peers { permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 } } peers { permanent_uuid: "666c7f486a9c4bdaaee9ea119edcd538" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39957 } }
I20260926 06:28:58.712770 21826 raft_consensus.cc:374] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:58.712790 21826 raft_consensus.cc:729] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64d6fdf3632646deb45f31c945847f28, State: Initialized, Role: FOLLOWER
I20260926 06:28:58.712872 21826 consensus_queue.cc:260] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 } } peers { permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 } } peers { permanent_uuid: "666c7f486a9c4bdaaee9ea119edcd538" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39957 } }
I20260926 06:28:58.713119 21826 ts_tablet_manager.cc:1428] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 06:28:58.713145 21678 heartbeater.cc:503] Master 127.2.39.254:37629 was elected leader, sending a full tablet report...
I20260926 06:28:58.713914 21827 tablet_bootstrap.cc:492] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c: Bootstrap starting.
I20260926 06:28:58.714093 21828 tablet_bootstrap.cc:492] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: No bootstrap required, opened a new log
I20260926 06:28:58.714207 21828 ts_tablet_manager.cc:1397] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 06:28:58.714725 21827 tablet_bootstrap.cc:654] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:58.715333 21827 log.cc:824] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:58.716162 21827 tablet_bootstrap.cc:492] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c: No bootstrap required, opened a new log
I20260926 06:28:58.716239 21827 ts_tablet_manager.cc:1397] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 06:28:58.716328 21828 raft_consensus.cc:348] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 } } peers { permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 } } peers { permanent_uuid: "666c7f486a9c4bdaaee9ea119edcd538" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39957 } }
I20260926 06:28:58.716497 21828 raft_consensus.cc:374] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:58.716538 21828 raft_consensus.cc:729] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666c7f486a9c4bdaaee9ea119edcd538, State: Initialized, Role: FOLLOWER
I20260926 06:28:58.716656 21828 consensus_queue.cc:260] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 } } peers { permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 } } peers { permanent_uuid: "666c7f486a9c4bdaaee9ea119edcd538" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39957 } }
I20260926 06:28:58.716954 21828 ts_tablet_manager.cc:1428] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:28:58.716957 21812 heartbeater.cc:503] Master 127.2.39.254:37629 was elected leader, sending a full tablet report...
I20260926 06:28:58.719218 21827 raft_consensus.cc:348] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 } } peers { permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 } } peers { permanent_uuid: "666c7f486a9c4bdaaee9ea119edcd538" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39957 } }
I20260926 06:28:58.719403 21827 raft_consensus.cc:374] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:58.719460 21827 raft_consensus.cc:729] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f43c6507bf464379a91e46dcd15f770c, State: Initialized, Role: FOLLOWER
I20260926 06:28:58.719581 21827 consensus_queue.cc:260] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 } } peers { permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 } } peers { permanent_uuid: "666c7f486a9c4bdaaee9ea119edcd538" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39957 } }
I20260926 06:28:58.719877 21827 ts_tablet_manager.cc:1428] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 06:28:58.719899 21544 heartbeater.cc:503] Master 127.2.39.254:37629 was elected leader, sending a full tablet report...
W20260926 06:28:58.757819 21679 tablet.cc:2369] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:28:58.858394 21833 raft_consensus.cc:482] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:58.858530 21833 raft_consensus.cc:504] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 } } peers { permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 } } peers { permanent_uuid: "666c7f486a9c4bdaaee9ea119edcd538" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39957 } }
I20260926 06:28:58.858943 21833 leader_election.cc:288] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f43c6507bf464379a91e46dcd15f770c (127.2.39.193:45811), 64d6fdf3632646deb45f31c945847f28 (127.2.39.194:42315)
W20260926 06:28:58.861444 21545 tablet.cc:2369] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:28:58.862840 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.859240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:45585 (local address 127.2.39.194:42315)
0926 06:28:58.859284 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.859286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.859507 (+ 221us) server_negotiation.cc:407] Connection header received
0926 06:28:58.859536 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.859537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.859548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.859620 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:58.859755 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.860431 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.861892 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.862485 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.862552 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.862561 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.862731 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.862772 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 06:28:58.862839 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.859244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:58647 (local address 127.2.39.193:45811)
0926 06:28:58.859293 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.859295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.859307 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:28:58.859356 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.859357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.859370 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.859424 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:58.859617 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.860392 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.861892 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.862470 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.862552 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.862561 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.862731 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.862772 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 06:28:58.862862 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.859145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45811 (local address 127.2.39.195:58647)
0926 06:28:58.859202 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:58.859216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:58.859280 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:58.859458 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:58.859459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:58.859461 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:58.859536 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.859538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.860498 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.860499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.861792 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.861795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.862564 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.862565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.862648 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.862649 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 06:28:58.862711 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:28:58.862765 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":54}
I20260926 06:28:58.862869 21835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.859145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:42315 (local address 127.2.39.195:45585)
0926 06:28:58.859398 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:58.859424 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:58.859491 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:58.859626 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:58.859627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:58.859628 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:58.859687 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.859688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.860518 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.860518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.861813 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:58.861815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.862564 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.862565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:58.862645 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.862649 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:58.862711 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:28:58.862766 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":157,"thread_start_us":118,"threads_started":1}
I20260926 06:28:58.863251 21632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae6cdef38584f74a44dcacb86a211c0" candidate_uuid: "666c7f486a9c4bdaaee9ea119edcd538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d6fdf3632646deb45f31c945847f28" is_pre_election: true
I20260926 06:28:58.863251 21498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae6cdef38584f74a44dcacb86a211c0" candidate_uuid: "666c7f486a9c4bdaaee9ea119edcd538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f43c6507bf464379a91e46dcd15f770c" is_pre_election: true
I20260926 06:28:58.863404 21498 raft_consensus.cc:2454] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666c7f486a9c4bdaaee9ea119edcd538 in term 0.
I20260926 06:28:58.863404 21632 raft_consensus.cc:2454] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 666c7f486a9c4bdaaee9ea119edcd538 in term 0.
I20260926 06:28:58.863623 21699 leader_election.cc:302] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d6fdf3632646deb45f31c945847f28, 666c7f486a9c4bdaaee9ea119edcd538; no voters:
I20260926 06:28:58.863741 21833 raft_consensus.cc:2781] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:28:58.863795 21833 raft_consensus.cc:482] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:28:58.863821 21833 raft_consensus.cc:3037] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:58.864530 21833 raft_consensus.cc:504] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 } } peers { permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 } } peers { permanent_uuid: "666c7f486a9c4bdaaee9ea119edcd538" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39957 } }
I20260926 06:28:58.864672 21833 leader_election.cc:288] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [CANDIDATE]: Term 1 election: Requested vote from peers f43c6507bf464379a91e46dcd15f770c (127.2.39.193:45811), 64d6fdf3632646deb45f31c945847f28 (127.2.39.194:42315)
I20260926 06:28:58.864826 21498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae6cdef38584f74a44dcacb86a211c0" candidate_uuid: "666c7f486a9c4bdaaee9ea119edcd538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f43c6507bf464379a91e46dcd15f770c"
I20260926 06:28:58.864827 21632 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae6cdef38584f74a44dcacb86a211c0" candidate_uuid: "666c7f486a9c4bdaaee9ea119edcd538" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64d6fdf3632646deb45f31c945847f28"
I20260926 06:28:58.864905 21498 raft_consensus.cc:3037] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:58.864907 21632 raft_consensus.cc:3037] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:58.865783 21498 raft_consensus.cc:2454] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666c7f486a9c4bdaaee9ea119edcd538 in term 1.
I20260926 06:28:58.865783 21632 raft_consensus.cc:2454] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 666c7f486a9c4bdaaee9ea119edcd538 in term 1.
I20260926 06:28:58.865975 21699 leader_election.cc:302] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64d6fdf3632646deb45f31c945847f28, 666c7f486a9c4bdaaee9ea119edcd538; no voters:
I20260926 06:28:58.866071 21833 raft_consensus.cc:2781] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:58.866222 21833 raft_consensus.cc:686] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 1 LEADER]: Becoming Leader. State: Replica: 666c7f486a9c4bdaaee9ea119edcd538, State: Running, Role: LEADER
I20260926 06:28:58.866312 21833 consensus_queue.cc:237] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 } } peers { permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 } } peers { permanent_uuid: "666c7f486a9c4bdaaee9ea119edcd538" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39957 } }
I20260926 06:28:58.867167 21353 catalog_manager.cc:5495] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 reported cstate change: term changed from 0 to 1, leader changed from <none> to 666c7f486a9c4bdaaee9ea119edcd538 (127.2.39.195). New cstate: current_term: 1 leader_uuid: "666c7f486a9c4bdaaee9ea119edcd538" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "666c7f486a9c4bdaaee9ea119edcd538" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 39957 } health_report { overall_health: HEALTHY } } }
I20260926 06:28:58.885026 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.882954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.2.39.195:39957)
0926 06:28:58.882995 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:58.882997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:58.883025 (+ 28us) server_negotiation.cc:407] Connection header received
0926 06:28:58.883073 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:58.883074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:58.883089 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:58.883142 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:28:58.883262 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.883718 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.884413 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:58.884596 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:58.884644 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:58.884766 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:28:58.884857 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:28:58.884918 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:58.884957 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 06:28:58.884972 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:28:58.886927 21498 raft_consensus.cc:1261] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c [term 1 FOLLOWER]: Refusing update from remote peer 666c7f486a9c4bdaaee9ea119edcd538: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:58.886927 21632 raft_consensus.cc:1261] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28 [term 1 FOLLOWER]: Refusing update from remote peer 666c7f486a9c4bdaaee9ea119edcd538: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:58.887326 21833 consensus_queue.cc:1035] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f43c6507bf464379a91e46dcd15f770c" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 45811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:58.887508 21836 consensus_queue.cc:1035] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64d6fdf3632646deb45f31c945847f28" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 42315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:58.889632 21839 mvcc.cc:205] Tried to move back new op lower bound from 7333495352877907968 to 7333495352797020160. Current Snapshot: MvccSnapshot[applied={T|T < 7333495352877907968}]
I20260926 06:28:58.889912 21840 mvcc.cc:205] Tried to move back new op lower bound from 7333495352877907968 to 7333495352797020160. Current Snapshot: MvccSnapshot[applied={T|T < 7333495352877907968}]
I20260926 06:28:58.890739 21841 mvcc.cc:205] Tried to move back new op lower bound from 7333495352877907968 to 7333495352797020160. Current Snapshot: MvccSnapshot[applied={T|T < 7333495352877907968}]
I20260926 06:28:58.892086 21353 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:58.892164 21353 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:58.892877 21353 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 206bf5b4d82244e8bc1f8b476c11a583: Sending DeleteTablet for 3 replicas of tablet 4ae6cdef38584f74a44dcacb86a211c0
I20260926 06:28:58.893200 21746 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae6cdef38584f74a44dcacb86a211c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:58 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:55852
I20260926 06:28:58.893187 21478 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae6cdef38584f74a44dcacb86a211c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:58 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:49346
I20260926 06:28:58.893227 21612 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae6cdef38584f74a44dcacb86a211c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:58 UTC) from {username='kudu', principal='kudu/127.2.39.254@KRBTEST.COM'} at 127.0.0.1:59516
I20260926 06:28:58.893400 21849 tablet_replica.cc:330] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c: stopping tablet replica
I20260926 06:28:58.893441 21851 tablet_replica.cc:330] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: stopping tablet replica
I20260926 06:28:58.893489 21849 raft_consensus.cc:2229] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:58.893514 21851 raft_consensus.cc:2229] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:58.893613 21849 raft_consensus.cc:2258] T 4ae6cdef38584f74a44dcacb86a211c0 P f43c6507bf464379a91e46dcd15f770c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:58.893613 21851 raft_consensus.cc:2258] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:58.893757 21850 tablet_replica.cc:330] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: stopping tablet replica
I20260926 06:28:58.893846 21850 raft_consensus.cc:2229] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:28:58.893951 21850 raft_consensus.cc:2258] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:58.894052 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 21415
I20260926 06:28:58.895051 21850 ts_tablet_manager.cc:1905] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:58.895215 21851 ts_tablet_manager.cc:1905] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:58.896981 21850 ts_tablet_manager.cc:1918] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:58.897048 21850 log.cc:1196] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-2/wal/wals/4ae6cdef38584f74a44dcacb86a211c0
I20260926 06:28:58.897310 21851 ts_tablet_manager.cc:1918] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:58.897378 21850 ts_tablet_manager.cc:1939] T 4ae6cdef38584f74a44dcacb86a211c0 P 666c7f486a9c4bdaaee9ea119edcd538: Deleting consensus metadata
I20260926 06:28:58.897492 21851 log.cc:1196] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/4ae6cdef38584f74a44dcacb86a211c0
I20260926 06:28:58.897807 21340 catalog_manager.cc:4841] TS 666c7f486a9c4bdaaee9ea119edcd538 (127.2.39.195:39957): tablet 4ae6cdef38584f74a44dcacb86a211c0 (table test-table [id=f77d85018bba473188a2e3187ef954d2]) successfully deleted
I20260926 06:28:58.897841 21851 ts_tablet_manager.cc:1939] T 4ae6cdef38584f74a44dcacb86a211c0 P 64d6fdf3632646deb45f31c945847f28: Deleting consensus metadata
I20260926 06:28:58.898245 21337 catalog_manager.cc:4841] TS 64d6fdf3632646deb45f31c945847f28 (127.2.39.194:42315): tablet 4ae6cdef38584f74a44dcacb86a211c0 (table test-table [id=f77d85018bba473188a2e3187ef954d2]) successfully deleted
W20260926 06:28:58.899755 21340 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:45811 (error 108)
I20260926 06:28:58.900242 21823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:58.900113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:45811 (local address 127.0.0.1:49352)
0926 06:28:58.900171 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:58.900205 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:45811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 06:28:58.900415 21340 catalog_manager.cc:4584] TS f43c6507bf464379a91e46dcd15f770c (127.2.39.193:45811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ae6cdef38584f74a44dcacb86a211c0: Network error: Client connection negotiation failed: client connection to 127.2.39.193:45811: connect: Connection refused (error 111)
I20260926 06:28:58.901736 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 21549
I20260926 06:28:58.906765 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 21683
I20260926 06:28:58.911422 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 21321
2026-09-26T06:28:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2023 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T06:28:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:28:58Z Disabled control of system clock
I20260926 06:28:58.938447 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:34181
--webserver_interface=127.2.39.254
--webserver_port=0
--builtin_ntp_servers=127.2.39.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:34181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:28:59.018414 21856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:59.018632 21856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:59.018651 21856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:59.019733 21856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:28:59.019778 21856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:59.019790 21856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:59.019804 21856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:28:59.019815 21856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:28:59.021070 21856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:34181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:34181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:59.021255 21856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:59.021481 21856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:59.026368 21862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:59.026340 21864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:59.026319 21861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:59.026700 21856 server_base.cc:1034] running on GCE node
I20260926 06:28:59.026983 21856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:59.027225 21856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:59.028359 21856 hybrid_clock.cc:648] HybridClock initialized: now 1790404139028339 us; error 36 us; skew 500 ppm
I20260926 06:28:59.029510 21856 webserver.cc:460] Webserver started at http://127.2.39.254:36557/ using document root <none> and password file <none>
I20260926 06:28:59.029765 21856 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:59.029810 21856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:59.029899 21856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:59.030928 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "0708e001996345fda4b9081565b0d735"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "b7137c3a8e96312c87480c8c53b22aa9"
server_key_iv: "648ad6ad251ce22d607ad2619e6a0e2e"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.031227 21856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "0708e001996345fda4b9081565b0d735"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "b7137c3a8e96312c87480c8c53b22aa9"
server_key_iv: "648ad6ad251ce22d607ad2619e6a0e2e"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.032413 21856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:28:59.033074 21870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:59.033258 21856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:28:59.033329 21856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "0708e001996345fda4b9081565b0d735"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "b7137c3a8e96312c87480c8c53b22aa9"
server_key_iv: "648ad6ad251ce22d607ad2619e6a0e2e"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.033414 21856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:59.049153 21856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:59.049443 21856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:59.049611 21856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:59.053289 21856 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:34181
I20260926 06:28:59.053336 21922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:34181 every 8 connection(s)
I20260926 06:28:59.053745 21856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:28:59.054668 21923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:59.056879 21923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735: Bootstrap starting.
I20260926 06:28:59.057531 21923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:59.057858 21923 log.cc:824] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:59.058558 21923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735: No bootstrap required, opened a new log
I20260926 06:28:59.060075 21923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0708e001996345fda4b9081565b0d735" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34181 } }
I20260926 06:28:59.060207 21923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:59.060230 21923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0708e001996345fda4b9081565b0d735, State: Initialized, Role: FOLLOWER
I20260926 06:28:59.060390 21923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0708e001996345fda4b9081565b0d735" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34181 } }
I20260926 06:28:59.060456 21923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:28:59.060494 21923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:28:59.060541 21923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:59.061038 21923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0708e001996345fda4b9081565b0d735" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34181 } }
I20260926 06:28:59.061142 21923 leader_election.cc:302] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0708e001996345fda4b9081565b0d735; no voters:
I20260926 06:28:59.061308 21923 leader_election.cc:288] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:28:59.061344 21926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:59.061520 21926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [term 1 LEADER]: Becoming Leader. State: Replica: 0708e001996345fda4b9081565b0d735, State: Running, Role: LEADER
I20260926 06:28:59.061561 21923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:28:59.061653 21926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0708e001996345fda4b9081565b0d735" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34181 } }
I20260926 06:28:59.062007 21928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0708e001996345fda4b9081565b0d735. Latest consensus state: current_term: 1 leader_uuid: "0708e001996345fda4b9081565b0d735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0708e001996345fda4b9081565b0d735" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34181 } } }
I20260926 06:28:59.062086 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0708e001996345fda4b9081565b0d735" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0708e001996345fda4b9081565b0d735" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34181 } } }
I20260926 06:28:59.062103 21928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:59.062150 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735 [sys.catalog]: This master's current role is: LEADER
I20260926 06:28:59.062407 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 21856
I20260926 06:28:59.062527 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:28:59.062580 21935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:28:59.062737 2207 external_mini_cluster.cc:1428] Setting key 9d395610a4bc1b06ad6226a679980083
I20260926 06:28:59.063073 21935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:28:59.066613 21935 catalog_manager.cc:1347] Generated new cluster ID: ea29d4ab5944458ba61da1815f8c7792
I20260926 06:28:59.066666 21935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:28:59.069399 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.063218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39258 (local address 127.2.39.254:34181)
0926 06:28:59.063410 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.063419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.063847 (+ 428us) server_negotiation.cc:407] Connection header received
0926 06:28:59.063962 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.063977 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.064045 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.064186 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:59.064340 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.065696 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.066899 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.067515 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.067629 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.068527 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:59.068531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:59.068531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:59.068730 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:59.068773 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:59.068788 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:59.068845 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.069110 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.069185 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260926 06:28:59.082737 21935 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:28:59.083420 21935 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:28:59.089313 21935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735: Generated new TSK 0
I20260926 06:28:59.089550 21935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:28:59.091487 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.193:0
--local_ip_for_outbound_sockets=127.2.39.193
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34181
--builtin_ntp_servers=127.2.39.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:28:59.171243 21947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:59.171469 21947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:59.171507 21947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:59.172715 21947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:59.172773 21947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:59.172817 21947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.193
I20260926 06:28:59.174263 21947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.39.193
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.193
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:59.174523 21947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:59.174815 21947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:59.181022 21953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:59.181022 21952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:59.181200 21955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:59.181229 21947 server_base.cc:1034] running on GCE node
I20260926 06:28:59.181474 21947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:59.181758 21947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:59.182926 21947 hybrid_clock.cc:648] HybridClock initialized: now 1790404139182908 us; error 32 us; skew 500 ppm
I20260926 06:28:59.184139 21947 webserver.cc:460] Webserver started at http://127.2.39.193:44853/ using document root <none> and password file <none>
I20260926 06:28:59.184396 21947 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:59.184478 21947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:59.184631 21947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:59.185779 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/data/instance:
uuid: "3e7577d70bd748069cec9c9935995675"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "55669749494254dfa7782a1cf4f369af"
server_key_iv: "33edf0c7045302bba05d130cc9638d16"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.186126 21947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance:
uuid: "3e7577d70bd748069cec9c9935995675"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "55669749494254dfa7782a1cf4f369af"
server_key_iv: "33edf0c7045302bba05d130cc9638d16"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.187467 21947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:28:59.188246 21961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:59.188412 21947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:28:59.188529 21947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/wal
uuid: "3e7577d70bd748069cec9c9935995675"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "55669749494254dfa7782a1cf4f369af"
server_key_iv: "33edf0c7045302bba05d130cc9638d16"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.188611 21947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:59.196645 21947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:59.196913 21947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:59.197057 21947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:59.197261 21947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:59.197664 21947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:59.197718 21947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:59.197757 21947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:59.197784 21947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:59.203437 21947 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.193:35305
I20260926 06:28:59.203501 22074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.193:35305 every 8 connection(s)
I20260926 06:28:59.203830 21947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/data/info.pb
I20260926 06:28:59.206187 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 21947
I20260926 06:28:59.206295 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-0/wal/instance
I20260926 06:28:59.206467 2207 external_mini_cluster.cc:1428] Setting key 7f4cbd6363687ef58d520036ded94385
I20260926 06:28:59.207408 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.194:0
--local_ip_for_outbound_sockets=127.2.39.194
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34181
--builtin_ntp_servers=127.2.39.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 06:28:59.209532 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.204542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.193:35637 (local address 127.2.39.254:34181)
0926 06:28:59.204615 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.204617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.204844 (+ 227us) server_negotiation.cc:407] Connection header received
0926 06:28:59.205300 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.205301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.205313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.205361 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:59.206072 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.206790 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.207987 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.208313 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.208389 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.209234 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:59.209236 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:59.209236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:59.209244 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:59.209262 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:59.209267 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:59.209337 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.209404 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.209433 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:28:59.209681 22077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.204556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:34181 (local address 127.2.39.193:35637)
0926 06:28:59.204730 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.204762 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:59.205219 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:59.205420 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:59.205424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:59.205433 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:59.205978 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.205982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.206865 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.206866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.207897 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.207900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.208398 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.208399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.208516 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.209097 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:28:59.209097 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:59.209125 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:28:59.209146 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:59.209337 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:59.209340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:59.209342 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:28:59.209387 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 06:28:59.209465 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260926 06:28:59.210225 22075 heartbeater.cc:347] Connected to a master server at 127.2.39.254:34181
I20260926 06:28:59.210311 22075 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:59.210494 22075 heartbeater.cc:511] Master 127.2.39.254:34181 requested a full tablet report, sending...
I20260926 06:28:59.210994 21887 ts_manager.cc:194] Registered new tserver with Master: 3e7577d70bd748069cec9c9935995675 (127.2.39.193:35305)
I20260926 06:28:59.211607 21887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.193:35637
W20260926 06:28:59.290484 22078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:59.290709 22078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:59.290752 22078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:59.292071 22078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:59.292127 22078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:59.292179 22078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.194
I20260926 06:28:59.293792 22078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.39.194
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.194
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:59.294062 22078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:59.294339 22078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:59.299257 22083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:59.299355 22078 server_base.cc:1034] running on GCE node
W20260926 06:28:59.299299 22086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:59.299278 22084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:59.299675 22078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:59.299870 22078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:59.301009 22078 hybrid_clock.cc:648] HybridClock initialized: now 1790404139300984 us; error 41 us; skew 500 ppm
I20260926 06:28:59.302312 22078 webserver.cc:460] Webserver started at http://127.2.39.194:41185/ using document root <none> and password file <none>
I20260926 06:28:59.302568 22078 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:59.302618 22078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:59.302770 22078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:59.303818 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/data/instance:
uuid: "a48c3b1e131849bfa39fd242ac0b7e30"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "0bc193fe1fc31d794b474fb07d73cf24"
server_key_iv: "538af3e7d23830590ff27eb175fa357c"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.304177 22078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance:
uuid: "a48c3b1e131849bfa39fd242ac0b7e30"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "0bc193fe1fc31d794b474fb07d73cf24"
server_key_iv: "538af3e7d23830590ff27eb175fa357c"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.305476 22078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:28:59.306222 22092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:59.306399 22078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:28:59.306494 22078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/wal
uuid: "a48c3b1e131849bfa39fd242ac0b7e30"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "0bc193fe1fc31d794b474fb07d73cf24"
server_key_iv: "538af3e7d23830590ff27eb175fa357c"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.306579 22078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:59.325541 22078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:59.325935 22078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:59.326086 22078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:59.326310 22078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:59.326632 22078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:59.326689 22078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:59.326730 22078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:59.326756 22078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:59.332485 22078 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.194:41927
I20260926 06:28:59.332531 22205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.194:41927 every 8 connection(s)
I20260926 06:28:59.332836 22078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/data/info.pb
I20260926 06:28:59.338140 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.333607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:41685 (local address 127.2.39.254:34181)
0926 06:28:59.333668 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.333671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.333867 (+ 196us) server_negotiation.cc:407] Connection header received
0926 06:28:59.334401 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.334402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.334414 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.334465 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:59.335140 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.335637 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.336693 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.336899 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.336986 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.337844 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:59.337845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:59.337846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:59.337850 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:59.337864 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:59.337868 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:59.337921 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.338034 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.338063 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:28:59.338347 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.333537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:34181 (local address 127.2.39.194:41685)
0926 06:28:59.333752 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.333785 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:59.334284 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:59.334532 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:59.334536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:59.334545 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:59.335045 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.335049 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.335753 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.335754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.336605 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.336607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.337002 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.337004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.337129 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.337708 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:28:59.337709 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:59.337737 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:28:59.337758 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:59.337929 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:59.337933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:59.337936 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:59.338035 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 06:28:59.338132 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":99,"threads_started":1}
I20260926 06:28:59.338836 22206 heartbeater.cc:347] Connected to a master server at 127.2.39.254:34181
I20260926 06:28:59.338943 22206 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:59.339125 22206 heartbeater.cc:511] Master 127.2.39.254:34181 requested a full tablet report, sending...
I20260926 06:28:59.339498 21887 ts_manager.cc:194] Registered new tserver with Master: a48c3b1e131849bfa39fd242ac0b7e30 (127.2.39.194:41927)
I20260926 06:28:59.339851 21887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.194:41685
I20260926 06:28:59.342872 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 22078
I20260926 06:28:59.342988 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/wal/instance
I20260926 06:28:59.343187 2207 external_mini_cluster.cc:1428] Setting key 21ebb9d435e93753616d659a5759e50e
I20260926 06:28:59.344048 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.39.195:0
--local_ip_for_outbound_sockets=127.2.39.195
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34181
--builtin_ntp_servers=127.2.39.212:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 06:28:59.423856 22209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:28:59.424048 22209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:28:59.424069 22209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:28:59.425215 22209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:28:59.425251 22209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:28:59.425276 22209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.39.195
I20260926 06:28:59.426662 22209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.39.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.39.195
--webserver_port=0
--tserver_master_addrs=127.2.39.254:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.39.195
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:28:59.426858 22209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:28:59.427140 22209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:28:59.432137 22214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:59.432114 22217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:28:59.432121 22215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:28:59.432153 22209 server_base.cc:1034] running on GCE node
I20260926 06:28:59.432513 22209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:28:59.432740 22209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:28:59.433892 22209 hybrid_clock.cc:648] HybridClock initialized: now 1790404139433876 us; error 28 us; skew 500 ppm
I20260926 06:28:59.435107 22209 webserver.cc:460] Webserver started at http://127.2.39.195:42105/ using document root <none> and password file <none>
I20260926 06:28:59.435330 22209 fs_manager.cc:359] Metadata directory not provided
I20260926 06:28:59.435396 22209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:28:59.435563 22209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:28:59.436671 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/data/instance:
uuid: "8ef823e3a5784c23a43df31060d1d166"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "312ff761b2be5c560a2852a74bfa0fdd"
server_key_iv: "eb8ee93a2e1a0c36baf8d78182d56b6e"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.437065 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance:
uuid: "8ef823e3a5784c23a43df31060d1d166"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "312ff761b2be5c560a2852a74bfa0fdd"
server_key_iv: "eb8ee93a2e1a0c36baf8d78182d56b6e"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.438421 22209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:28:59.439127 22223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:59.439285 22209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:28:59.439381 22209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/wal
uuid: "8ef823e3a5784c23a43df31060d1d166"
format_stamp: "Formatted at 2026-09-26 06:28:59 on dist-test-slave-fkbn"
server_key: "312ff761b2be5c560a2852a74bfa0fdd"
server_key_iv: "eb8ee93a2e1a0c36baf8d78182d56b6e"
server_key_version: "encryptionkey@0"
I20260926 06:28:59.439456 22209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:28:59.450831 22209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:28:59.451112 22209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:28:59.451260 22209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:28:59.451478 22209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 06:28:59.451781 22209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 06:28:59.451834 22209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:59.451874 22209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 06:28:59.451901 22209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 06:28:59.457496 22209 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.195:38471
I20260926 06:28:59.457643 22336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.195:38471 every 8 connection(s)
I20260926 06:28:59.457937 22209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/data/info.pb
I20260926 06:28:59.458820 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 22209
I20260926 06:28:59.458937 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-2/wal/instance
I20260926 06:28:59.459115 2207 external_mini_cluster.cc:1428] Setting key 1b05dd4b9894767c2002788d61d025f7
I20260926 06:28:59.463162 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.458666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.195:37397 (local address 127.2.39.254:34181)
0926 06:28:59.458706 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.458708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.458937 (+ 229us) server_negotiation.cc:407] Connection header received
0926 06:28:59.459578 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.459579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.459594 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.459653 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:59.460415 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.460895 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.462002 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.462196 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.462249 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.462965 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:59.462966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:59.462966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:59.462970 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:59.462982 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:59.462986 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:59.463027 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.463084 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.463107 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:28:59.463377 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.458611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:34181 (local address 127.2.39.195:37397)
0926 06:28:59.458814 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.458847 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:59.459459 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:59.459724 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:59.459728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:59.459737 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:28:59.460338 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.460343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.460987 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.460988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.461934 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.461937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.462261 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.462262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.462364 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.462840 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 06:28:59.462840 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:28:59.462868 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 06:28:59.462887 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:28:59.463027 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:28:59.463030 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:28:59.463032 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 06:28:59.463077 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 06:28:59.463147 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260926 06:28:59.463896 22337 heartbeater.cc:347] Connected to a master server at 127.2.39.254:34181
I20260926 06:28:59.463984 22337 heartbeater.cc:464] Registering TS with master...
I20260926 06:28:59.464176 22337 heartbeater.cc:511] Master 127.2.39.254:34181 requested a full tablet report, sending...
I20260926 06:28:59.464556 21887 ts_manager.cc:194] Registered new tserver with Master: 8ef823e3a5784c23a43df31060d1d166 (127.2.39.195:38471)
I20260926 06:28:59.464910 21887 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.39.195:37397
I20260926 06:28:59.470795 2207 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 06:28:59.474769 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.472080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.2.39.254:34181)
0926 06:28:59.472148 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.472150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.472229 (+ 79us) server_negotiation.cc:407] Connection header received
0926 06:28:59.472263 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.472264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.472274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.472313 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:28:59.472448 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.472885 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.473725 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.473903 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.473966 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.474509 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:28:59.474510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:28:59.474510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:28:59.474513 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:28:59.474525 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:28:59.474529 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:28:59.474571 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.474630 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.474650 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:28:59.476189 21887 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39264:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 06:28:59.477699 21887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 06:28:59.483739 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.479755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:38471 (local address 127.0.0.1:37568)
0926 06:28:59.480027 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.480067 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:59.480136 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:59.480375 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:59.480378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:59.480380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:59.480454 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.480457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.481418 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.481419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.482723 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.482726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.483457 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.483458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.483530 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.483542 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 06:28:59.483607 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 06:28:59.483655 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":5,"thread_start_us":86,"threads_started":1}
I20260926 06:28:59.483836 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.479917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.2.39.194:41927)
0926 06:28:59.480235 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.480239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.480248 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:28:59.480285 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.480286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.480300 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.480379 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:59.480545 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.481704 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.482788 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.483484 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.483592 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.483676 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.483707 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.483748 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":80,"threads_started":1}
I20260926 06:28:59.483845 22345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.479755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:41927 (local address 127.0.0.1:57766)
0926 06:28:59.480074 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.480084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:59.480136 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:59.480374 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:59.480379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:59.480382 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:59.480462 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.480464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.481809 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.481810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.482716 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.482718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.483600 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.483601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.483652 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.483655 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:59.483695 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 06:28:59.483725 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":3,"thread_start_us":60,"threads_started":1}
I20260926 06:28:59.484071 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.479920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37568 (local address 127.2.39.195:38471)
0926 06:28:59.480129 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.480135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.480152 (+ 17us) server_negotiation.cc:407] Connection header received
0926 06:28:59.480230 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.480233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.480249 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.480538 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:59.480591 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.481330 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.482815 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.483360 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.483449 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.483537 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.483925 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.483970 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":106,"threads_started":1}
I20260926 06:28:59.484079 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.479771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:35305 (local address 127.0.0.1:59324)
0926 06:28:59.480246 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.480255 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:59.480290 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:59.480526 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:59.480527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:59.480529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:59.480567 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.480568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.481825 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.481833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.482903 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.482904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.483850 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.483851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.483919 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.483923 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 06:28:59.483987 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 06:28:59.484024 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":25,"thread_start_us":90,"threads_started":1}
I20260926 06:28:59.484380 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.480019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.2.39.193:35305)
0926 06:28:59.480302 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.480306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.480316 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:28:59.480356 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.480358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.480374 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.480447 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:59.480644 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.481744 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.482987 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.483679 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.483743 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.483814 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.484257 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.484303 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":78,"threads_started":1}
I20260926 06:28:59.485044 22271 tablet_service.cc:1467] Processing CreateTablet for tablet f02a20c37c244021891cc9621bc0fafb (DEFAULT_TABLE table=test-table [id=85d68b40996044c3ad7c7f737e372ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:59.485036 22140 tablet_service.cc:1467] Processing CreateTablet for tablet f02a20c37c244021891cc9621bc0fafb (DEFAULT_TABLE table=test-table [id=85d68b40996044c3ad7c7f737e372ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:59.485029 22009 tablet_service.cc:1467] Processing CreateTablet for tablet f02a20c37c244021891cc9621bc0fafb (DEFAULT_TABLE table=test-table [id=85d68b40996044c3ad7c7f737e372ece]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 06:28:59.485378 22271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02a20c37c244021891cc9621bc0fafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:59.485378 22009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02a20c37c244021891cc9621bc0fafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:59.485378 22140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02a20c37c244021891cc9621bc0fafb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:28:59.487921 22351 tablet_bootstrap.cc:492] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166: Bootstrap starting.
I20260926 06:28:59.488543 22351 tablet_bootstrap.cc:654] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:59.488583 22352 tablet_bootstrap.cc:492] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: Bootstrap starting.
I20260926 06:28:59.488816 22353 tablet_bootstrap.cc:492] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675: Bootstrap starting.
I20260926 06:28:59.489018 22351 log.cc:824] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:59.489226 22352 tablet_bootstrap.cc:654] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:59.489454 22353 tablet_bootstrap.cc:654] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675: Neither blocks nor log segments found. Creating new log.
I20260926 06:28:59.489540 22352 log.cc:824] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:59.489801 22353 log.cc:824] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675: Log is configured to *not* fsync() on all Append() calls
I20260926 06:28:59.489910 22351 tablet_bootstrap.cc:492] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166: No bootstrap required, opened a new log
I20260926 06:28:59.489976 22351 ts_tablet_manager.cc:1397] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:28:59.490238 22352 tablet_bootstrap.cc:492] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: No bootstrap required, opened a new log
I20260926 06:28:59.490289 22352 ts_tablet_manager.cc:1397] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 06:28:59.492010 22351 raft_consensus.cc:348] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41927 } } peers { permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 } } peers { permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 } }
I20260926 06:28:59.492158 22351 raft_consensus.cc:374] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:59.492202 22351 raft_consensus.cc:729] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ef823e3a5784c23a43df31060d1d166, State: Initialized, Role: FOLLOWER
I20260926 06:28:59.492318 22351 consensus_queue.cc:260] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41927 } } peers { permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 } } peers { permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 } }
I20260926 06:28:59.492401 22352 raft_consensus.cc:348] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41927 } } peers { permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 } } peers { permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 } }
I20260926 06:28:59.492566 22352 raft_consensus.cc:374] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:59.492616 22351 ts_tablet_manager.cc:1428] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 06:28:59.492624 22352 raft_consensus.cc:729] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48c3b1e131849bfa39fd242ac0b7e30, State: Initialized, Role: FOLLOWER
I20260926 06:28:59.492693 22337 heartbeater.cc:503] Master 127.2.39.254:34181 was elected leader, sending a full tablet report...
I20260926 06:28:59.492736 22352 consensus_queue.cc:260] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41927 } } peers { permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 } } peers { permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 } }
I20260926 06:28:59.493000 22352 ts_tablet_manager.cc:1428] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 06:28:59.493060 22206 heartbeater.cc:503] Master 127.2.39.254:34181 was elected leader, sending a full tablet report...
I20260926 06:28:59.493603 22353 tablet_bootstrap.cc:492] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675: No bootstrap required, opened a new log
I20260926 06:28:59.493671 22353 ts_tablet_manager.cc:1397] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 06:28:59.495684 22353 raft_consensus.cc:348] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41927 } } peers { permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 } } peers { permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 } }
I20260926 06:28:59.495815 22353 raft_consensus.cc:374] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:28:59.495858 22353 raft_consensus.cc:729] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7577d70bd748069cec9c9935995675, State: Initialized, Role: FOLLOWER
I20260926 06:28:59.495970 22353 consensus_queue.cc:260] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41927 } } peers { permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 } } peers { permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 } }
I20260926 06:28:59.496208 22353 ts_tablet_manager.cc:1428] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 06:28:59.496261 22075 heartbeater.cc:503] Master 127.2.39.254:34181 was elected leader, sending a full tablet report...
W20260926 06:28:59.583361 22207 tablet.cc:2369] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:28:59.700628 22358 raft_consensus.cc:482] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 06:28:59.700747 22358 raft_consensus.cc:504] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41927 } } peers { permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 } } peers { permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 } }
I20260926 06:28:59.701188 22358 leader_election.cc:288] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ef823e3a5784c23a43df31060d1d166 (127.2.39.195:38471), 3e7577d70bd748069cec9c9935995675 (127.2.39.193:35305)
I20260926 06:28:59.703977 22208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.701360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:38471 (local address 127.2.39.194:54725)
0926 06:28:59.701434 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.701447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:59.701493 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:59.701686 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:59.701687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:59.701688 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:59.701763 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.701765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.702394 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.702395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.703188 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.703190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.703728 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.703729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.703798 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.703801 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:59.703861 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 06:28:59.703901 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 06:28:59.703977 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.701427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:54725 (local address 127.2.39.195:38471)
0926 06:28:59.701487 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.701489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.701501 (+ 12us) server_negotiation.cc:407] Connection header received
0926 06:28:59.701595 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.701596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.701607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.701682 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:59.701841 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.702315 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.703273 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.703634 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.703728 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.703736 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.703877 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.703916 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 06:28:59.704067 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.701472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.39.194:50467 (local address 127.2.39.193:35305)
0926 06:28:59.701517 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.701519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.701702 (+ 183us) server_negotiation.cc:407] Connection header received
0926 06:28:59.701770 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.701771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.701783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.701855 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 06:28:59.702006 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.702453 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.703382 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.703723 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.703805 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.703812 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.703961 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.704005 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 06:28:59.704304 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.701386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:35305 (local address 127.2.39.194:50467)
0926 06:28:59.701625 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.701637 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 06:28:59.701714 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:28:59.701858 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:28:59.701858 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:28:59.701860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 06:28:59.701932 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.701934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.702535 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.702536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.703292 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:28:59.703294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.703816 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.703817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:28:59.703890 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.703893 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 06:28:59.703956 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 06:28:59.703993 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":95,"threads_started":1}
I20260926 06:28:59.704313 22291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02a20c37c244021891cc9621bc0fafb" candidate_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef823e3a5784c23a43df31060d1d166" is_pre_election: true
I20260926 06:28:59.704480 22291 raft_consensus.cc:2454] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a48c3b1e131849bfa39fd242ac0b7e30 in term 0.
W20260926 06:28:59.704706 22076 tablet.cc:2369] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:28:59.704741 22029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02a20c37c244021891cc9621bc0fafb" candidate_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e7577d70bd748069cec9c9935995675" is_pre_election: true
I20260926 06:28:59.704916 22029 raft_consensus.cc:2454] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a48c3b1e131849bfa39fd242ac0b7e30 in term 0.
I20260926 06:28:59.704977 22093 leader_election.cc:302] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ef823e3a5784c23a43df31060d1d166, a48c3b1e131849bfa39fd242ac0b7e30; no voters:
I20260926 06:28:59.705116 22358 raft_consensus.cc:2781] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 06:28:59.705168 22358 raft_consensus.cc:482] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 06:28:59.705184 22358 raft_consensus.cc:3037] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:59.705912 22358 raft_consensus.cc:504] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41927 } } peers { permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 } } peers { permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 } }
I20260926 06:28:59.706056 22358 leader_election.cc:288] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [CANDIDATE]: Term 1 election: Requested vote from peers 8ef823e3a5784c23a43df31060d1d166 (127.2.39.195:38471), 3e7577d70bd748069cec9c9935995675 (127.2.39.193:35305)
I20260926 06:28:59.706192 22291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02a20c37c244021891cc9621bc0fafb" candidate_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ef823e3a5784c23a43df31060d1d166"
I20260926 06:28:59.706252 22029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02a20c37c244021891cc9621bc0fafb" candidate_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e7577d70bd748069cec9c9935995675"
I20260926 06:28:59.706292 22291 raft_consensus.cc:3037] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:59.706365 22029 raft_consensus.cc:3037] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:28:59.707060 22029 raft_consensus.cc:2454] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a48c3b1e131849bfa39fd242ac0b7e30 in term 1.
I20260926 06:28:59.707072 22291 raft_consensus.cc:2454] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a48c3b1e131849bfa39fd242ac0b7e30 in term 1.
I20260926 06:28:59.707237 22094 leader_election.cc:302] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e7577d70bd748069cec9c9935995675, a48c3b1e131849bfa39fd242ac0b7e30; no voters:
I20260926 06:28:59.707335 22358 raft_consensus.cc:2781] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:28:59.707487 22358 raft_consensus.cc:686] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 1 LEADER]: Becoming Leader. State: Replica: a48c3b1e131849bfa39fd242ac0b7e30, State: Running, Role: LEADER
I20260926 06:28:59.707589 22358 consensus_queue.cc:237] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41927 } } peers { permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 } } peers { permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 } }
W20260926 06:28:59.708453 22338 tablet.cc:2369] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 06:28:59.708422 21887 catalog_manager.cc:5495] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 reported cstate change: term changed from 0 to 1, leader changed from <none> to a48c3b1e131849bfa39fd242ac0b7e30 (127.2.39.194). New cstate: current_term: 1 leader_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a48c3b1e131849bfa39fd242ac0b7e30" member_type: VOTER last_known_addr { host: "127.2.39.194" port: 41927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 } health_report { overall_health: UNKNOWN } } }
I20260926 06:28:59.764845 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.762630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.2.39.194:41927)
0926 06:28:59.762696 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 06:28:59.762697 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:28:59.762707 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:28:59.762780 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:28:59.762781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:28:59.762799 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:28:59.762863 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:28:59.762989 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.763522 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.764216 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:28:59.764393 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:28:59.764440 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:28:59.764567 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:28:59.764662 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:28:59.764720 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:28:59.764750 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 06:28:59.764764 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 06:28:59.766706 22291 raft_consensus.cc:1261] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166 [term 1 FOLLOWER]: Refusing update from remote peer a48c3b1e131849bfa39fd242ac0b7e30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:59.766719 22029 raft_consensus.cc:1261] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675 [term 1 FOLLOWER]: Refusing update from remote peer a48c3b1e131849bfa39fd242ac0b7e30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:28:59.767108 22358 consensus_queue.cc:1035] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ef823e3a5784c23a43df31060d1d166" member_type: VOTER last_known_addr { host: "127.2.39.195" port: 38471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:59.767235 22361 consensus_queue.cc:1035] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e7577d70bd748069cec9c9935995675" member_type: VOTER last_known_addr { host: "127.2.39.193" port: 35305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:28:59.769511 22364 mvcc.cc:205] Tried to move back new op lower bound from 7333495356481798144 to 7333495356242804736. Current Snapshot: MvccSnapshot[applied={T|T < 7333495356481798144}]
I20260926 06:28:59.770323 22365 mvcc.cc:205] Tried to move back new op lower bound from 7333495356481798144 to 7333495356242804736. Current Snapshot: MvccSnapshot[applied={T|T < 7333495356481798144}]
I20260926 06:28:59.771607 21887 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:59.771692 21887 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39264:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 06:28:59.772429 21887 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0708e001996345fda4b9081565b0d735: Sending DeleteTablet for 3 replicas of tablet f02a20c37c244021891cc9621bc0fafb
I20260926 06:28:59.772738 22140 tablet_service.cc:1514] Processing DeleteTablet for tablet f02a20c37c244021891cc9621bc0fafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:59 UTC) from {username='slave'} at 127.0.0.1:57766
I20260926 06:28:59.772771 22009 tablet_service.cc:1514] Processing DeleteTablet for tablet f02a20c37c244021891cc9621bc0fafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:59 UTC) from {username='slave'} at 127.0.0.1:59324
I20260926 06:28:59.772873 22271 tablet_service.cc:1514] Processing DeleteTablet for tablet f02a20c37c244021891cc9621bc0fafb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 06:28:59 UTC) from {username='slave'} at 127.0.0.1:37568
I20260926 06:28:59.772977 22375 tablet_replica.cc:330] T f02a20c37c244021891cc9621bc0fafb P 3e7577d70bd748069cec9c9935995675: stopping tablet replica
I20260926 06:28:59.773170 22374 tablet_replica.cc:330] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: stopping tablet replica
I20260926 06:28:59.773253 22374 raft_consensus.cc:2229] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 1 LEADER]: Raft consensus shutting down.
I20260926 06:28:59.773314 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 21947
I20260926 06:28:59.774959 22376 tablet_replica.cc:330] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166: stopping tablet replica
I20260926 06:28:59.775079 22376 raft_consensus.cc:2229] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 06:28:59.775183 22376 raft_consensus.cc:2258] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:59.773368 22374 raft_consensus.cc:2258] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 06:28:59.777916 22376 ts_tablet_manager.cc:1905] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 06:28:59.778316 22374 ts_tablet_manager.cc:1905] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 06:28:59.779369 21872 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.39.193:35305 (error 108)
I20260926 06:28:59.779867 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 22078
I20260926 06:28:59.780066 22374 ts_tablet_manager.cc:1918] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:59.780273 22374 log.cc:1196] T f02a20c37c244021891cc9621bc0fafb P a48c3b1e131849bfa39fd242ac0b7e30: Deleting WAL directory at /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790403899360596-2207-0/minicluster-data/ts-1/wal/wals/f02a20c37c244021891cc9621bc0fafb
I20260926 06:28:59.780696 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.779998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.193:35305 (local address 127.0.0.1:59330)
0926 06:28:59.780629 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.780669 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.193:35305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614}
W20260926 06:28:59.780820 21872 catalog_manager.cc:4584] TS 3e7577d70bd748069cec9c9935995675 (127.2.39.193:35305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f02a20c37c244021891cc9621bc0fafb: Network error: Client connection negotiation failed: client connection to 127.2.39.193:35305: connect: Connection refused (error 111)
I20260926 06:28:59.785285 22376 ts_tablet_manager.cc:1918] T f02a20c37c244021891cc9621bc0fafb P 8ef823e3a5784c23a43df31060d1d166: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 06:28:59.785285 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 22209
I20260926 06:28:59.785342 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.785224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.194:41927 (local address 127.0.0.1:57782)
0926 06:28:59.785280 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.785311 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.194:41927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 06:28:59.789634 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 21856
I20260926 06:28:59.789795 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:28:59.789667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.195:38471 (local address 127.0.0.1:37572)
0926 06:28:59.789728 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:28:59.789759 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.195:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
2026-09-26T06:28:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (878 ms)
[----------] 6 tests from EncryptionPolicyTest (8407 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:28:59 dist-test-slave-fkbn krb5kdc[22380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:28:59 dist-test-slave-fkbn krb5kdc[22380](info): set up 2 sockets
Sep 26 06:28:59 dist-test-slave-fkbn krb5kdc[22380](info): commencing operation
krb5kdc: starting...
W20260926 06:29:00.829545 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:00 dist-test-slave-fkbn krb5kdc[22380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:00.948576 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33429
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:01.031415 22393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:01.031621 22393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:01.031646 22393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:01.032930 22393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:01.032979 22393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:01.032995 22393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:01.033010 22393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:01.033051 22393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:01.034566 22393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33429
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:01.034857 22393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:01.035116 22393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:01.040320 22398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:01.040321 22401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:01.040443 22399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:01.040596 22393 server_base.cc:1034] running on GCE node
I20260926 06:29:01.040824 22393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:01.041113 22393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:01.042279 22393 hybrid_clock.cc:648] HybridClock initialized: now 1790404141042261 us; error 39 us; skew 500 ppm
Sep 26 06:29:01 dist-test-slave-fkbn krb5kdc[22380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:01.044620 22393 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:01.044988 22393 webserver.cc:460] Webserver started at http://127.0.0.1:40927/ using document root <none> and password file <none>
I20260926 06:29:01.045209 22393 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:01.045253 22393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:01.045395 22393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:01.046479 22393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "6160224be84c48fb9945bda85e13e80a"
format_stamp: "Formatted at 2026-09-26 06:29:01 on dist-test-slave-fkbn"
server_key: "d033a4c1544cf9643780adfa94274d04"
server_key_iv: "b5b022e277d5693ad6d36492979dc163"
server_key_version: "encryptionkey@0"
I20260926 06:29:01.046826 22393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "6160224be84c48fb9945bda85e13e80a"
format_stamp: "Formatted at 2026-09-26 06:29:01 on dist-test-slave-fkbn"
server_key: "d033a4c1544cf9643780adfa94274d04"
server_key_iv: "b5b022e277d5693ad6d36492979dc163"
server_key_version: "encryptionkey@0"
I20260926 06:29:01.048110 22393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:29:01.048822 22408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:01.048987 22393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:29:01.049083 22393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "6160224be84c48fb9945bda85e13e80a"
format_stamp: "Formatted at 2026-09-26 06:29:01 on dist-test-slave-fkbn"
server_key: "d033a4c1544cf9643780adfa94274d04"
server_key_iv: "b5b022e277d5693ad6d36492979dc163"
server_key_version: "encryptionkey@0"
I20260926 06:29:01.049160 22393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:01.059388 22393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:01.061546 22393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:01.061781 22393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:01.065461 22393 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33429
I20260926 06:29:01.065534 22460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33429 every 8 connection(s)
I20260926 06:29:01.065879 22393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:01.066790 22461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:01.068851 22461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a: Bootstrap starting.
I20260926 06:29:01.069375 22461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:01.069682 22461 log.cc:824] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:01.070313 22461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a: No bootstrap required, opened a new log
I20260926 06:29:01.071787 22461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6160224be84c48fb9945bda85e13e80a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } }
I20260926 06:29:01.071918 22461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:01.071938 22461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6160224be84c48fb9945bda85e13e80a, State: Initialized, Role: FOLLOWER
I20260926 06:29:01.072034 22461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6160224be84c48fb9945bda85e13e80a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } }
I20260926 06:29:01.072115 22461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:01.072150 22461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:01.072206 22461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:01.072718 22461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6160224be84c48fb9945bda85e13e80a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } }
I20260926 06:29:01.072834 22461 leader_election.cc:302] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6160224be84c48fb9945bda85e13e80a; no voters:
I20260926 06:29:01.073036 22461 leader_election.cc:288] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:01.073060 22464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:01.073139 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 22393
I20260926 06:29:01.073174 22464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [term 1 LEADER]: Becoming Leader. State: Replica: 6160224be84c48fb9945bda85e13e80a, State: Running, Role: LEADER
I20260926 06:29:01.073256 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:01.073243 22464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6160224be84c48fb9945bda85e13e80a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } }
I20260926 06:29:01.073293 22461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:01.073485 2207 external_mini_cluster.cc:1428] Setting key fa198eeb7e66d34e1daa87d0be0d672e
I20260926 06:29:01.073535 22464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6160224be84c48fb9945bda85e13e80a. Latest consensus state: current_term: 1 leader_uuid: "6160224be84c48fb9945bda85e13e80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6160224be84c48fb9945bda85e13e80a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } } }
I20260926 06:29:01.073546 22465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6160224be84c48fb9945bda85e13e80a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6160224be84c48fb9945bda85e13e80a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33429 } } }
I20260926 06:29:01.073719 22465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:01.073634 22464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:01.074326 22468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:01.074862 22468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:01.076306 22468 catalog_manager.cc:1347] Generated new cluster ID: ac6e7ada28734c6889b587d8c00c403d
I20260926 06:29:01.076356 22468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:29:01 dist-test-slave-fkbn krb5kdc[22380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 06:29:01.084322 22468 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:01.084965 22468 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:01.085690 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:01.074158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36393 (local address 127.0.0.1:33429)
0926 06:29:01.074595 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:01.074603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:01.074626 (+ 23us) server_negotiation.cc:407] Connection header received
0926 06:29:01.074767 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:01.074774 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:01.074841 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:01.075802 (+ 961us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:01.075892 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:01.077231 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:01.078276 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:01.078902 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:01.079025 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:01.081864 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:01.081868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:01.081868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:01.081879 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:01.083897 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:01.084382 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:01.084385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:01.084386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:01.084496 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:01.084759 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:01.084760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:01.084760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:01.085057 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:01.085207 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:01.085325 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 06:29:01.085432 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":90,"threads_started":1}
I20260926 06:29:01.086683 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:01.091279 22468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6160224be84c48fb9945bda85e13e80a: Generated new TSK 0
I20260926 06:29:01.091461 22468 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:01.095268 22475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:01.088192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41559 (local address 127.0.0.1:33429)
0926 06:29:01.088226 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:01.088228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:01.088300 (+ 72us) server_negotiation.cc:407] Connection header received
0926 06:29:01.088354 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:01.088355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:01.088365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:01.088429 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:01.088994 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:01.089813 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:01.090850 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:01.091165 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:01.091231 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:01.092730 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:01.092731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:01.092731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:01.092746 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:01.094168 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:01.094566 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:01.094567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:01.094567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:01.094616 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:01.094838 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:01.094839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:01.094839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:01.094929 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:01.095020 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:01.095167 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 06:29:01.095213 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:29:01.097514 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 22393
W20260926 06:29:01.102276 22378 connection.cc:535] client connection to 127.0.0.1:33429 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33429): Connection reset by peer (error 104)
2026-09-26T06:29:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:01 dist-test-slave-fkbn krb5kdc[22492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:01 dist-test-slave-fkbn krb5kdc[22492](info): set up 2 sockets
Sep 26 06:29:01 dist-test-slave-fkbn krb5kdc[22492](info): commencing operation
krb5kdc: starting...
W20260926 06:29:02.133541 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:02 dist-test-slave-fkbn krb5kdc[22492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:02.244802 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:02.323566 22505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:02.323765 22505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:02.323787 22505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:02.324978 22505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:02.325019 22505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:02.325033 22505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:02.325083 22505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:02.325101 22505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:02.326557 22505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41521
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:02.326783 22505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:02.326982 22505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:02.331816 22510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:02.331825 22511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:02.331916 22505 server_base.cc:1034] running on GCE node
W20260926 06:29:02.331825 22513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:02.332224 22505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:02.332542 22505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:02.333704 22505 hybrid_clock.cc:648] HybridClock initialized: now 1790404142333692 us; error 41 us; skew 500 ppm
Sep 26 06:29:02 dist-test-slave-fkbn krb5kdc[22492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:02.336037 22505 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:02.336472 22505 webserver.cc:460] Webserver started at http://127.0.0.1:41821/ using document root <none> and password file <none>
I20260926 06:29:02.336647 22505 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:02.336679 22505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:02.336768 22505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:02.337846 22505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9"
format_stamp: "Formatted at 2026-09-26 06:29:02 on dist-test-slave-fkbn"
server_key: "49c27ff7a8e0e05743547e290f00848f"
server_key_iv: "3a021bc7c4939535619cf56be3a4562f"
server_key_version: "encryptionkey@0"
I20260926 06:29:02.338151 22505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9"
format_stamp: "Formatted at 2026-09-26 06:29:02 on dist-test-slave-fkbn"
server_key: "49c27ff7a8e0e05743547e290f00848f"
server_key_iv: "3a021bc7c4939535619cf56be3a4562f"
server_key_version: "encryptionkey@0"
I20260926 06:29:02.339392 22505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:29:02.340059 22520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:02.340265 22505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:29:02.340332 22505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9"
format_stamp: "Formatted at 2026-09-26 06:29:02 on dist-test-slave-fkbn"
server_key: "49c27ff7a8e0e05743547e290f00848f"
server_key_iv: "3a021bc7c4939535619cf56be3a4562f"
server_key_version: "encryptionkey@0"
I20260926 06:29:02.340427 22505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:02.352074 22505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:02.353749 22505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:02.353889 22505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:02.357762 22505 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41521
I20260926 06:29:02.357810 22572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41521 every 8 connection(s)
I20260926 06:29:02.358107 22505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:02.358747 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 22505
I20260926 06:29:02.358879 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:02.359074 2207 external_mini_cluster.cc:1428] Setting key 63e855dd82caca7d697e5403252aaea5
I20260926 06:29:02.359246 22573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:02.362041 22573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9: Bootstrap starting.
I20260926 06:29:02.362593 22573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:02.362884 22573 log.cc:824] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:02.363663 22573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9: No bootstrap required, opened a new log
I20260926 06:29:02.364300 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:02.359577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39975 (local address 127.0.0.1:41521)
0926 06:29:02.359766 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:02.359774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:02.359797 (+ 23us) server_negotiation.cc:407] Connection header received
0926 06:29:02.360270 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:02.360275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:02.360317 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:02.360464 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:02.360682 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:02.361549 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:02.362408 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:02.362882 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:02.363018 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:02.363648 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:02.363651 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:02.363652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:02.363838 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:02.363866 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:02.363877 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:02.363944 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:02.364059 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 06:29:02.364102 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260926 06:29:02.365290 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:02.366050 22573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } }
I20260926 06:29:02.366195 22573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:02.366238 22573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac3eaf8248a4d258dcd59e1c57f6ae9, State: Initialized, Role: FOLLOWER
I20260926 06:29:02.366356 22573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } }
I20260926 06:29:02.366456 22573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:02.366493 22573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:02.366546 22573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:02.367082 22573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } }
I20260926 06:29:02.367194 22573 leader_election.cc:302] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ac3eaf8248a4d258dcd59e1c57f6ae9; no voters:
I20260926 06:29:02.367378 22573 leader_election.cc:288] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:02.367419 22583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:02.367553 22583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [term 1 LEADER]: Becoming Leader. State: Replica: 4ac3eaf8248a4d258dcd59e1c57f6ae9, State: Running, Role: LEADER
I20260926 06:29:02.367614 22573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:02.367640 22583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } }
I20260926 06:29:02.367921 22583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ac3eaf8248a4d258dcd59e1c57f6ae9. Latest consensus state: current_term: 1 leader_uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } } }
I20260926 06:29:02.367992 22583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:02.368196 22588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:02.368654 22588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:02.369172 22585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ac3eaf8248a4d258dcd59e1c57f6ae9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41521 } } }
I20260926 06:29:02.369493 22585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:02.370230 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:02.366669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35609 (local address 127.0.0.1:41521)
0926 06:29:02.366728 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:02.366730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:02.366867 (+ 137us) server_negotiation.cc:407] Connection header received
0926 06:29:02.366912 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:02.366913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:02.366922 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:02.366976 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:02.367117 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:02.367813 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:02.368939 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:02.369216 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:02.369266 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:02.369987 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:02.369989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:02.369989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:02.369993 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:02.370006 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:02.370009 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:02.370063 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:02.370132 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 06:29:02.370140 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 06:29:02.370771 22588 catalog_manager.cc:1347] Generated new cluster ID: 72bdb585f97f41dd89733cdbc452fa92
I20260926 06:29:02.370820 22588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:02.378997 22588 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:02.379650 22588 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:02.391548 22588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ac3eaf8248a4d258dcd59e1c57f6ae9: Generated new TSK 0
I20260926 06:29:02.391707 22588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:02.395448 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 22505
W20260926 06:29:02.399626 22490 connection.cc:535] client connection to 127.0.0.1:41521 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41521): Connection reset by peer (error 104)
2026-09-26T06:29:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:02 dist-test-slave-fkbn krb5kdc[22604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:02 dist-test-slave-fkbn krb5kdc[22604](info): set up 2 sockets
Sep 26 06:29:02 dist-test-slave-fkbn krb5kdc[22604](info): commencing operation
krb5kdc: starting...
W20260926 06:29:03.425992 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:03 dist-test-slave-fkbn krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:03.540014 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:03.621616 22617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:03.621801 22617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:03.621824 22617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:03.622978 22617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:03.623021 22617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:03.623036 22617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:03.623049 22617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:03.623060 22617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:03.624603 22617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38697
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:03.624835 22617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:03.625083 22617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:03.629931 22622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:03.629956 22625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:03.629959 22623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:03.630100 22617 server_base.cc:1034] running on GCE node
I20260926 06:29:03.630515 22617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:03.630748 22617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:03.631880 22617 hybrid_clock.cc:648] HybridClock initialized: now 1790404143631870 us; error 49 us; skew 500 ppm
Sep 26 06:29:03 dist-test-slave-fkbn krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404143, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:03.634300 22617 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:03.634708 22617 webserver.cc:460] Webserver started at http://127.0.0.1:43167/ using document root <none> and password file <none>
I20260926 06:29:03.634944 22617 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:03.635011 22617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:03.635178 22617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:03.636179 22617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "16aa7b214182416790d7265eee9e7e76"
format_stamp: "Formatted at 2026-09-26 06:29:03 on dist-test-slave-fkbn"
server_key: "11decf864c10335e5490ec0942d5dcde"
server_key_iv: "c711f581cb2c0c41edb832c71e02df8b"
server_key_version: "encryptionkey@0"
I20260926 06:29:03.636528 22617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "16aa7b214182416790d7265eee9e7e76"
format_stamp: "Formatted at 2026-09-26 06:29:03 on dist-test-slave-fkbn"
server_key: "11decf864c10335e5490ec0942d5dcde"
server_key_iv: "c711f581cb2c0c41edb832c71e02df8b"
server_key_version: "encryptionkey@0"
I20260926 06:29:03.637856 22617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:29:03.638552 22632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:03.638717 22617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:29:03.638813 22617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "16aa7b214182416790d7265eee9e7e76"
format_stamp: "Formatted at 2026-09-26 06:29:03 on dist-test-slave-fkbn"
server_key: "11decf864c10335e5490ec0942d5dcde"
server_key_iv: "c711f581cb2c0c41edb832c71e02df8b"
server_key_version: "encryptionkey@0"
I20260926 06:29:03.638892 22617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:03.642079 22617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:03.643764 22617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:03.643920 22617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:03.647778 22617 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38697
I20260926 06:29:03.647819 22684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38697 every 8 connection(s)
I20260926 06:29:03.648141 22617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:03.649019 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:03.651221 22685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76: Bootstrap starting.
I20260926 06:29:03.651765 22685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:03.652040 22685 log.cc:824] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:03.652755 22685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76: No bootstrap required, opened a new log
I20260926 06:29:03.654140 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 22617
I20260926 06:29:03.654261 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:03.654453 2207 external_mini_cluster.cc:1428] Setting key 3bf4e5ac663a19747ebac62368fff6f4
I20260926 06:29:03.654443 22685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16aa7b214182416790d7265eee9e7e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38697 } }
I20260926 06:29:03.654636 22685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:03.654685 22685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16aa7b214182416790d7265eee9e7e76, State: Initialized, Role: FOLLOWER
I20260926 06:29:03.654826 22685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16aa7b214182416790d7265eee9e7e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38697 } }
I20260926 06:29:03.655227 22685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:03.655269 22685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:03.655320 22685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:03.656481 22685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16aa7b214182416790d7265eee9e7e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38697 } }
I20260926 06:29:03.656606 22685 leader_election.cc:302] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16aa7b214182416790d7265eee9e7e76; no voters:
I20260926 06:29:03.656796 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:03.654902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34337 (local address 127.0.0.1:38697)
0926 06:29:03.655133 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:03.655141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:03.655185 (+ 44us) server_negotiation.cc:407] Connection header received
0926 06:29:03.655333 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:03.655341 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:03.655369 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:03.655501 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:03.656131 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:03.656134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:03.656134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:03.656137 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:03.656172 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:03.656183 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:03.656236 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:03.656408 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 06:29:03.656467 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":141,"threads_started":1}
I20260926 06:29:03.656808 22685 leader_election.cc:288] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:03.657124 22685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:03.656862 22690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:03.657405 22690 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [term 1 LEADER]: Becoming Leader. State: Replica: 16aa7b214182416790d7265eee9e7e76, State: Running, Role: LEADER
I20260926 06:29:03.657532 22690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16aa7b214182416790d7265eee9e7e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38697 } }
I20260926 06:29:03.657842 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:03.658073 22690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16aa7b214182416790d7265eee9e7e76. Latest consensus state: current_term: 1 leader_uuid: "16aa7b214182416790d7265eee9e7e76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16aa7b214182416790d7265eee9e7e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38697 } } }
I20260926 06:29:03.658160 22690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:03.658293 22691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16aa7b214182416790d7265eee9e7e76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16aa7b214182416790d7265eee9e7e76" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38697 } } }
I20260926 06:29:03.658389 22691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76 [sys.catalog]: This master's current role is: LEADER
W20260926 06:29:03.658871 22705 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:29:03.658978 22705 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:03.659072 22702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:03.659231 22702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:03.661142 22702 catalog_manager.cc:1347] Generated new cluster ID: 769fcac2bde3458e8fd74a7069212ea5
I20260926 06:29:03.661218 22702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:03.661291 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:03.659721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56861 (local address 127.0.0.1:38697)
0926 06:29:03.659762 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:03.659764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:03.660000 (+ 236us) server_negotiation.cc:407] Connection header received
0926 06:29:03.660091 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:03.660092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:03.660098 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:03.660159 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:03.660938 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:03.660939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:03.660940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:03.660945 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:03.660959 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:03.660965 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:03.661002 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:03.661142 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 06:29:03.661158 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:29:03.665930 22702 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:03.666085 22702 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:03.668861 22702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16aa7b214182416790d7265eee9e7e76: Generated new TSK 0
I20260926 06:29:03.669014 22702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:03.674752 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 22617
2026-09-26T06:29:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1279 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:03 dist-test-slave-fkbn krb5kdc[22716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:03 dist-test-slave-fkbn krb5kdc[22716](info): set up 2 sockets
Sep 26 06:29:03 dist-test-slave-fkbn krb5kdc[22716](info): commencing operation
krb5kdc: starting...
W20260926 06:29:04.765641 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:04 dist-test-slave-fkbn krb5kdc[22716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:04.881100 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:04.959623 22729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:04.959860 22729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:04.959897 22729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:04.961036 22729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:04.961108 22729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:04.961133 22729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:04.961161 22729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:04.961187 22729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:04.962661 22729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:04.962957 22729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:04.963181 22729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:29:04.968327 22729 server_base.cc:1034] running on GCE node
W20260926 06:29:04.968302 22737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:04.968310 22735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:04.968310 22734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:04.968865 22729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:04.969151 22729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:04.970314 22729 hybrid_clock.cc:648] HybridClock initialized: now 1790404144970309 us; error 36 us; skew 500 ppm
Sep 26 06:29:04 dist-test-slave-fkbn krb5kdc[22716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:04.972653 22729 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:04.973031 22729 webserver.cc:460] Webserver started at http://127.0.0.1:39913/ using document root <none> and password file <none>
I20260926 06:29:04.973222 22729 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:04.973261 22729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:04.973346 22729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:04.974395 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "3bc5d964f0044b4d8673ded3e64f9f9a"
format_stamp: "Formatted at 2026-09-26 06:29:04 on dist-test-slave-fkbn"
server_key: "6bd350b034dfee6adfffc8ba969d3c02"
server_key_iv: "03576aa043839419b6fb217a50caf144"
server_key_version: "encryptionkey@0"
I20260926 06:29:04.974685 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "3bc5d964f0044b4d8673ded3e64f9f9a"
format_stamp: "Formatted at 2026-09-26 06:29:04 on dist-test-slave-fkbn"
server_key: "6bd350b034dfee6adfffc8ba969d3c02"
server_key_iv: "03576aa043839419b6fb217a50caf144"
server_key_version: "encryptionkey@0"
I20260926 06:29:04.975910 22729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:29:04.976575 22744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:04.976740 22729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:29:04.976818 22729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "3bc5d964f0044b4d8673ded3e64f9f9a"
format_stamp: "Formatted at 2026-09-26 06:29:04 on dist-test-slave-fkbn"
server_key: "6bd350b034dfee6adfffc8ba969d3c02"
server_key_iv: "03576aa043839419b6fb217a50caf144"
server_key_version: "encryptionkey@0"
I20260926 06:29:04.976886 22729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:04.984229 22729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:04.986397 22729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:04.986569 22729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:04.990307 22729 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38355
I20260926 06:29:04.990356 22796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38355 every 8 connection(s)
I20260926 06:29:04.990698 22729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:04.991671 22797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:04.993873 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a: Bootstrap starting.
I20260926 06:29:04.994413 22797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:04.994696 22797 log.cc:824] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:04.995384 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a: No bootstrap required, opened a new log
I20260926 06:29:04.995455 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 22729
I20260926 06:29:04.995585 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:04.995795 2207 external_mini_cluster.cc:1428] Setting key 41f97a9a1ef5c440f5d5e290bcb71628
I20260926 06:29:04.997212 22797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc5d964f0044b4d8673ded3e64f9f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } }
I20260926 06:29:04.997361 22797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:04.997426 22797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bc5d964f0044b4d8673ded3e64f9f9a, State: Initialized, Role: FOLLOWER
I20260926 06:29:04.997586 22797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc5d964f0044b4d8673ded3e64f9f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } }
I20260926 06:29:04.997689 22797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:04.997733 22797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:04.997782 22797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:04.998577 22797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc5d964f0044b4d8673ded3e64f9f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } }
I20260926 06:29:04.998694 22797 leader_election.cc:302] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bc5d964f0044b4d8673ded3e64f9f9a; no voters:
I20260926 06:29:04.998899 22797 leader_election.cc:288] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:04.998924 22802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:04.999037 22802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [term 1 LEADER]: Becoming Leader. State: Replica: 3bc5d964f0044b4d8673ded3e64f9f9a, State: Running, Role: LEADER
I20260926 06:29:04.999157 22797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:04.999168 22802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc5d964f0044b4d8673ded3e64f9f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } }
I20260926 06:29:04.999511 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bc5d964f0044b4d8673ded3e64f9f9a. Latest consensus state: current_term: 1 leader_uuid: "3bc5d964f0044b4d8673ded3e64f9f9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc5d964f0044b4d8673ded3e64f9f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } } }
I20260926 06:29:04.999518 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bc5d964f0044b4d8673ded3e64f9f9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bc5d964f0044b4d8673ded3e64f9f9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38355 } } }
I20260926 06:29:04.999682 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:04.999693 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:04.999955 22811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:05.000527 22811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 06:29:05 dist-test-slave-fkbn krb5kdc[22716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 06:29:05.006037 22811 catalog_manager.cc:1347] Generated new cluster ID: 3ab754d8622341a886ecf63ecdc705f9
I20260926 06:29:05.006091 22811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:05.007253 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:04.996246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45301 (local address 127.0.0.1:38355)
0926 06:29:04.996395 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:04.996405 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:04.996453 (+ 48us) server_negotiation.cc:407] Connection header received
0926 06:29:04.997197 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:04.997203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:04.997247 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:04.997378 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:04.998152 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:04.999255 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:05.000345 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:05.001003 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:05.001133 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:05.003636 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:05.003640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:05.003640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:05.003651 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:05.005297 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:05.005686 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:05.005689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:05.005689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:05.005805 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:05.006059 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:05.006060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:05.006060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:05.006351 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:05.006851 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:05.006917 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 06:29:05.007055 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":72,"threads_started":1}
I20260926 06:29:05.008252 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:05.020864 22801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:05.009710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57065 (local address 127.0.0.1:38355)
0926 06:29:05.009744 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:05.009746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:05.013775 (+ 4029us) server_negotiation.cc:407] Connection header received
0926 06:29:05.013867 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:05.013868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:05.013880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:05.013909 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:05.014608 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:05.015356 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:05.016392 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:05.016700 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:05.016760 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:05.018217 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:05.018218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:05.018219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:05.018225 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:05.019746 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:05.020121 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:05.020122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:05.020123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:05.020172 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:05.020372 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:05.020372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:05.020373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:05.020486 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:05.020567 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:05.020700 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 06:29:05.020769 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:29:05.029234 22811 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:05.029888 22811 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:05.037443 22811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bc5d964f0044b4d8673ded3e64f9f9a: Generated new TSK 0
I20260926 06:29:05.037642 22811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:05.040215 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 22729
2026-09-26T06:29:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:05 dist-test-slave-fkbn krb5kdc[22828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:05 dist-test-slave-fkbn krb5kdc[22828](info): set up 2 sockets
Sep 26 06:29:05 dist-test-slave-fkbn krb5kdc[22828](info): commencing operation
krb5kdc: starting...
W20260926 06:29:06.081542 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:06 dist-test-slave-fkbn krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:06.199940 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:06.279739 22841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:06.279935 22841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:06.279956 22841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:06.281025 22841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:06.281061 22841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:06.281075 22841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:06.281088 22841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:06.281100 22841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:06.282589 22841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:06.282773 22841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:06.282932 22841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:06.287644 22846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:06.287657 22847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:06.287657 22849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:06.287992 22841 server_base.cc:1034] running on GCE node
I20260926 06:29:06.288233 22841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:06.288522 22841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:06.289628 22841 hybrid_clock.cc:648] HybridClock initialized: now 1790404146289611 us; error 29 us; skew 500 ppm
Sep 26 06:29:06 dist-test-slave-fkbn krb5kdc[22828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:06.292171 22841 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:06.292685 22841 webserver.cc:460] Webserver started at http://127.0.0.1:34459/ using document root <none> and password file <none>
I20260926 06:29:06.292872 22841 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:06.292917 22841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:06.293007 22841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:06.294085 22841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "955467ca732b4ed2a595e9a5fcc53719"
format_stamp: "Formatted at 2026-09-26 06:29:06 on dist-test-slave-fkbn"
server_key: "cd36c32b25bb04ec15110ee8fe5e4ba2"
server_key_iv: "730065ec4b38d00d5a31a5a1ee2d946c"
server_key_version: "encryptionkey@0"
I20260926 06:29:06.294449 22841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "955467ca732b4ed2a595e9a5fcc53719"
format_stamp: "Formatted at 2026-09-26 06:29:06 on dist-test-slave-fkbn"
server_key: "cd36c32b25bb04ec15110ee8fe5e4ba2"
server_key_iv: "730065ec4b38d00d5a31a5a1ee2d946c"
server_key_version: "encryptionkey@0"
I20260926 06:29:06.295758 22841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:29:06.296430 22856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:06.296631 22841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:29:06.296730 22841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "955467ca732b4ed2a595e9a5fcc53719"
format_stamp: "Formatted at 2026-09-26 06:29:06 on dist-test-slave-fkbn"
server_key: "cd36c32b25bb04ec15110ee8fe5e4ba2"
server_key_iv: "730065ec4b38d00d5a31a5a1ee2d946c"
server_key_version: "encryptionkey@0"
I20260926 06:29:06.296782 22841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:06.315084 22841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:06.317404 22841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:06.317564 22841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:06.321401 22841 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33041
I20260926 06:29:06.321455 22908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33041 every 8 connection(s)
I20260926 06:29:06.321825 22841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:06.322799 22909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:06.324450 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 22841
I20260926 06:29:06.324575 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:06.324759 2207 external_mini_cluster.cc:1428] Setting key e71ce9010f912ec63f3b24c2d4746188
I20260926 06:29:06.325416 22909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719: Bootstrap starting.
I20260926 06:29:06.326022 22909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:06.326297 22909 log.cc:824] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:06.327028 22909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719: No bootstrap required, opened a new log
I20260926 06:29:06.329137 22909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955467ca732b4ed2a595e9a5fcc53719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } }
I20260926 06:29:06.329298 22909 raft_consensus.cc:374] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:06.329335 22909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955467ca732b4ed2a595e9a5fcc53719, State: Initialized, Role: FOLLOWER
I20260926 06:29:06.329483 22909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955467ca732b4ed2a595e9a5fcc53719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } }
I20260926 06:29:06.329572 22909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:06.329644 22909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:06.329689 22909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:06.330317 22909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955467ca732b4ed2a595e9a5fcc53719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } }
I20260926 06:29:06.330459 22909 leader_election.cc:302] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 955467ca732b4ed2a595e9a5fcc53719; no voters:
I20260926 06:29:06.330632 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:06.325258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49413 (local address 127.0.0.1:33041)
0926 06:29:06.325430 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:06.325438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:06.325461 (+ 23us) server_negotiation.cc:407] Connection header received
0926 06:29:06.325981 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:06.325987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:06.326034 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:06.326195 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:06.326416 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:06.327741 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:06.328481 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:06.329161 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:06.329269 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:06.329947 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:06.329950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:06.329950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:06.330105 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:06.330133 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:06.330143 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:06.330193 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:06.330292 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 06:29:06.330339 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260926 06:29:06.330662 22909 leader_election.cc:288] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:06.330693 22914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:06.330797 22914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [term 1 LEADER]: Becoming Leader. State: Replica: 955467ca732b4ed2a595e9a5fcc53719, State: Running, Role: LEADER
I20260926 06:29:06.330895 22909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:06.331516 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:06.330892 22914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955467ca732b4ed2a595e9a5fcc53719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } }
I20260926 06:29:06.332551 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955467ca732b4ed2a595e9a5fcc53719. Latest consensus state: current_term: 1 leader_uuid: "955467ca732b4ed2a595e9a5fcc53719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955467ca732b4ed2a595e9a5fcc53719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } } }
I20260926 06:29:06.332752 22915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "955467ca732b4ed2a595e9a5fcc53719" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955467ca732b4ed2a595e9a5fcc53719" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33041 } } }
I20260926 06:29:06.332980 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:06.332993 22915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719 [sys.catalog]: This master's current role is: LEADER
W20260926 06:29:06.333348 22930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:29:06.333418 22930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:06.333519 22934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:06.333736 22934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:06.335235 22934 catalog_manager.cc:1347] Generated new cluster ID: ed6882b39da6411e92ae25e48334a869
I20260926 06:29:06.335283 22934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:06.336656 22912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:06.333301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40071 (local address 127.0.0.1:33041)
0926 06:29:06.333339 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:06.333340 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:06.333712 (+ 372us) server_negotiation.cc:407] Connection header received
0926 06:29:06.333769 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:06.333770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:06.333791 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:06.333851 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:06.333968 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:06.334493 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:06.335482 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:06.335764 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:06.335827 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:06.336443 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:06.336444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:06.336444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:06.336448 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:06.336461 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:06.336465 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:06.336512 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:06.336572 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 06:29:06.336591 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:29:06.360540 22934 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:06.361075 22934 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:06.367139 22934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 955467ca732b4ed2a595e9a5fcc53719: Generated new TSK 0
I20260926 06:29:06.367349 22934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:06.370448 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 22841
2026-09-26T06:29:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:06 dist-test-slave-fkbn krb5kdc[22940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:06 dist-test-slave-fkbn krb5kdc[22940](info): set up 2 sockets
Sep 26 06:29:06 dist-test-slave-fkbn krb5kdc[22940](info): commencing operation
krb5kdc: starting...
W20260926 06:29:07.417817 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:07 dist-test-slave-fkbn krb5kdc[22940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:07.534484 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:07.620189 22953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:07.620404 22953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:07.620443 22953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:07.621639 22953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:07.621691 22953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:07.621717 22953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:07.621743 22953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:07.621765 22953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:07.623204 22953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:07.623473 22953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:07.623696 22953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:07.628439 22961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:07.628539 22958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:07.628471 22959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:07.628631 22953 server_base.cc:1034] running on GCE node
I20260926 06:29:07.628890 22953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:07.629173 22953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:07.630326 22953 hybrid_clock.cc:648] HybridClock initialized: now 1790404147630321 us; error 35 us; skew 500 ppm
Sep 26 06:29:07 dist-test-slave-fkbn krb5kdc[22940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:07.632676 22953 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:07.633087 22953 webserver.cc:460] Webserver started at http://127.0.0.1:43545/ using document root <none> and password file <none>
I20260926 06:29:07.633322 22953 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:07.633383 22953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:07.633525 22953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:07.634548 22953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "d7236c5491b740b6a2d6478346205cff"
format_stamp: "Formatted at 2026-09-26 06:29:07 on dist-test-slave-fkbn"
server_key: "91b117fc2f1027055e9f1d95a8b2475e"
server_key_iv: "7a565e01997b3a5678b402d76f2a8490"
server_key_version: "encryptionkey@0"
I20260926 06:29:07.634891 22953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "d7236c5491b740b6a2d6478346205cff"
format_stamp: "Formatted at 2026-09-26 06:29:07 on dist-test-slave-fkbn"
server_key: "91b117fc2f1027055e9f1d95a8b2475e"
server_key_iv: "7a565e01997b3a5678b402d76f2a8490"
server_key_version: "encryptionkey@0"
I20260926 06:29:07.636157 22953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 06:29:07.636816 22968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:07.636981 22953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:29:07.637073 22953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "d7236c5491b740b6a2d6478346205cff"
format_stamp: "Formatted at 2026-09-26 06:29:07 on dist-test-slave-fkbn"
server_key: "91b117fc2f1027055e9f1d95a8b2475e"
server_key_iv: "7a565e01997b3a5678b402d76f2a8490"
server_key_version: "encryptionkey@0"
I20260926 06:29:07.637145 22953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:07.640322 22953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:07.642464 22953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:07.642607 22953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:07.646276 22953 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46289
I20260926 06:29:07.646307 23020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46289 every 8 connection(s)
I20260926 06:29:07.646605 22953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:07.647471 23021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:07.649574 23021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff: Bootstrap starting.
I20260926 06:29:07.650174 23021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:07.650449 23021 log.cc:824] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:07.651094 23021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff: No bootstrap required, opened a new log
I20260926 06:29:07.652107 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 22953
I20260926 06:29:07.652230 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:07.652402 2207 external_mini_cluster.cc:1428] Setting key bb9b3dd6053a0d2f74b537bf82986d74
I20260926 06:29:07.652961 23021 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7236c5491b740b6a2d6478346205cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46289 } }
I20260926 06:29:07.653156 23021 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:07.653192 23021 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7236c5491b740b6a2d6478346205cff, State: Initialized, Role: FOLLOWER
I20260926 06:29:07.653307 23021 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7236c5491b740b6a2d6478346205cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46289 } }
I20260926 06:29:07.653393 23021 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:07.653472 23021 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:07.653532 23021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:07.654467 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:07.652879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43713 (local address 127.0.0.1:46289)
0926 06:29:07.653086 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:07.653095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:07.653126 (+ 31us) server_negotiation.cc:407] Connection header received
0926 06:29:07.653234 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:07.653241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:07.653270 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:07.653416 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:07.654045 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:07.654049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:07.654049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:07.654054 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:07.654109 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:07.654126 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:07.654197 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:07.654233 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 06:29:07.654270 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":100,"threads_started":1}
I20260926 06:29:07.654575 23021 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7236c5491b740b6a2d6478346205cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46289 } }
I20260926 06:29:07.654672 23021 leader_election.cc:302] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7236c5491b740b6a2d6478346205cff; no voters:
I20260926 06:29:07.654956 23021 leader_election.cc:288] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:07.654986 23026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:07.655148 23026 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [term 1 LEADER]: Becoming Leader. State: Replica: d7236c5491b740b6a2d6478346205cff, State: Running, Role: LEADER
I20260926 06:29:07.655224 23026 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7236c5491b740b6a2d6478346205cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46289 } }
I20260926 06:29:07.655274 23021 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:07.655339 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:07.655788 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7236c5491b740b6a2d6478346205cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7236c5491b740b6a2d6478346205cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46289 } } }
I20260926 06:29:07.655927 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:07.656104 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7236c5491b740b6a2d6478346205cff. Latest consensus state: current_term: 1 leader_uuid: "d7236c5491b740b6a2d6478346205cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7236c5491b740b6a2d6478346205cff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46289 } } }
I20260926 06:29:07.656209 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:07.656570 23033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:07.657140 23033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:07.658165 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:07.656819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50417 (local address 127.0.0.1:46289)
0926 06:29:07.656857 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:07.656859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:07.657073 (+ 214us) server_negotiation.cc:407] Connection header received
0926 06:29:07.657169 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:07.657170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:07.657176 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:07.657208 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:07.657957 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:07.657958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:07.657959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:07.657963 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:07.657976 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:07.657979 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:07.658029 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:07.658082 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 06:29:07.658095 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:29:07.658896 23033 catalog_manager.cc:1347] Generated new cluster ID: 328a972636fb41cf988f5d4a170fbf43
I20260926 06:29:07.658946 23033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:07.664044 23033 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:07.664206 23033 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:07.668932 23033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7236c5491b740b6a2d6478346205cff: Generated new TSK 0
I20260926 06:29:07.669114 23033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:07.673609 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 22953
2026-09-26T06:29:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:07 dist-test-slave-fkbn krb5kdc[23052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:07 dist-test-slave-fkbn krb5kdc[23052](info): set up 2 sockets
Sep 26 06:29:07 dist-test-slave-fkbn krb5kdc[23052](info): commencing operation
krb5kdc: starting...
W20260926 06:29:08.705729 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:08 dist-test-slave-fkbn krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:08.832355 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:08.911553 23065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:08.911787 23065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:08.911824 23065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:08.912953 23065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:08.913017 23065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:08.913043 23065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:08.913071 23065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:08.913094 23065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:08.914598 23065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:08.914870 23065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:08.915099 23065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:29:08.920029 23065 server_base.cc:1034] running on GCE node
W20260926 06:29:08.920030 23071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:08.919988 23070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:08.919968 23073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:08.920435 23065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:08.920682 23065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:08.921818 23065 hybrid_clock.cc:648] HybridClock initialized: now 1790404148921808 us; error 33 us; skew 500 ppm
Sep 26 06:29:08 dist-test-slave-fkbn krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:08.924201 23065 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:08.924633 23065 webserver.cc:460] Webserver started at http://127.0.0.1:34087/ using document root <none> and password file <none>
I20260926 06:29:08.924827 23065 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:08.924868 23065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:08.924963 23065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:08.926026 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "ad52ce10fe8e4dd89bbc913b558c19c0"
format_stamp: "Formatted at 2026-09-26 06:29:08 on dist-test-slave-fkbn"
server_key: "d4954f93931682de56fa60105d128c5f"
server_key_iv: "a2d163ec968e673a5f75c936b129aced"
server_key_version: "encryptionkey@0"
I20260926 06:29:08.926321 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "ad52ce10fe8e4dd89bbc913b558c19c0"
format_stamp: "Formatted at 2026-09-26 06:29:08 on dist-test-slave-fkbn"
server_key: "d4954f93931682de56fa60105d128c5f"
server_key_iv: "a2d163ec968e673a5f75c936b129aced"
server_key_version: "encryptionkey@0"
I20260926 06:29:08.927515 23065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:29:08.928157 23080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:08.928328 23065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:29:08.928406 23065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "ad52ce10fe8e4dd89bbc913b558c19c0"
format_stamp: "Formatted at 2026-09-26 06:29:08 on dist-test-slave-fkbn"
server_key: "d4954f93931682de56fa60105d128c5f"
server_key_iv: "a2d163ec968e673a5f75c936b129aced"
server_key_version: "encryptionkey@0"
I20260926 06:29:08.928459 23065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:08.936867 23065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:08.938688 23065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:08.938792 23065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:08.942549 23065 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33019
I20260926 06:29:08.942605 23132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33019 every 8 connection(s)
I20260926 06:29:08.942929 23065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:08.943883 23133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:08.946182 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0: Bootstrap starting.
I20260926 06:29:08.946259 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 23065
I20260926 06:29:08.946408 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:08.946606 2207 external_mini_cluster.cc:1428] Setting key febf65b9b93ca8f47cd04a3a7738a675
I20260926 06:29:08.946867 23133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:08.947212 23133 log.cc:824] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:08.947880 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0: No bootstrap required, opened a new log
I20260926 06:29:08.949704 23133 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad52ce10fe8e4dd89bbc913b558c19c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33019 } }
I20260926 06:29:08.949831 23133 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:08.949853 23133 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad52ce10fe8e4dd89bbc913b558c19c0, State: Initialized, Role: FOLLOWER
I20260926 06:29:08.949994 23133 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad52ce10fe8e4dd89bbc913b558c19c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33019 } }
I20260926 06:29:08.950052 23133 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:08.950098 23133 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:08.950145 23133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:08.950644 23133 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad52ce10fe8e4dd89bbc913b558c19c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33019 } }
I20260926 06:29:08.950755 23133 leader_election.cc:302] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad52ce10fe8e4dd89bbc913b558c19c0; no voters:
I20260926 06:29:08.950947 23133 leader_election.cc:288] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:08.950968 23138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:08.951082 23138 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [term 1 LEADER]: Becoming Leader. State: Replica: ad52ce10fe8e4dd89bbc913b558c19c0, State: Running, Role: LEADER
I20260926 06:29:08.951161 23138 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad52ce10fe8e4dd89bbc913b558c19c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33019 } }
I20260926 06:29:08.951200 23133 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:08.951514 23138 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad52ce10fe8e4dd89bbc913b558c19c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad52ce10fe8e4dd89bbc913b558c19c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33019 } } }
I20260926 06:29:08.951574 23138 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:08.951717 23138 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad52ce10fe8e4dd89bbc913b558c19c0. Latest consensus state: current_term: 1 leader_uuid: "ad52ce10fe8e4dd89bbc913b558c19c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad52ce10fe8e4dd89bbc913b558c19c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33019 } } }
I20260926 06:29:08.951776 23138 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0 [sys.catalog]: This master's current role is: LEADER
W20260926 06:29:08.952886 23154 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:29:08.952950 23154 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:08.953037 23143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:08.953210 23143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 06:29:08 dist-test-slave-fkbn krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 06:29:08.955163 23143 catalog_manager.cc:1347] Generated new cluster ID: 31f5f4e2495e4a2f96cab646c451fb76
I20260926 06:29:08.955219 23143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:08.957749 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:08.947072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:35981 (local address 127.0.0.1:33019)
0926 06:29:08.947259 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:08.947270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:08.947305 (+ 35us) server_negotiation.cc:407] Connection header received
0926 06:29:08.947859 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:08.947866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:08.947936 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:08.948090 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:08.948713 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:08.949636 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:08.950562 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:08.951097 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:08.951260 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:08.953998 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:08.954002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:08.954003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:08.954014 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:08.956015 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:08.956463 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:08.956466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:08.956466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:08.956560 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:08.956797 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:08.956798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:08.956798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:08.957111 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:08.957251 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:08.957405 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:29:08.957515 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260926 06:29:08.958571 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:08.967031 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:08.960111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:46037 (local address 127.0.0.1:33019)
0926 06:29:08.960159 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:08.960160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:08.960284 (+ 124us) server_negotiation.cc:407] Connection header received
0926 06:29:08.960314 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:08.960315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:08.960325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:08.960394 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:08.960935 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:08.961691 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:08.962647 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:08.962951 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:08.963026 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:08.964327 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:08.964328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:08.964329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:08.964335 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:08.965888 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:08.966266 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:08.966267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:08.966267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:08.966323 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:08.966535 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:08.966535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:08.966536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:08.966659 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:08.966761 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:08.966871 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 06:29:08.966933 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 06:29:08.969440 23143 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:08.970166 23143 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:08.978011 23143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad52ce10fe8e4dd89bbc913b558c19c0: Generated new TSK 0
I20260926 06:29:08.978225 23143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:08.980131 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 23065
W20260926 06:29:08.984413 23050 connection.cc:535] client connection to 127.0.0.1:33019 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33019): Connection reset by peer (error 104)
2026-09-26T06:29:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:09 dist-test-slave-fkbn krb5kdc[23164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:09 dist-test-slave-fkbn krb5kdc[23164](info): set up 2 sockets
Sep 26 06:29:09 dist-test-slave-fkbn krb5kdc[23164](info): commencing operation
krb5kdc: starting...
W20260926 06:29:10.009629 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:10 dist-test-slave-fkbn krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:10.124222 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:10.203765 23177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:10.203969 23177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:10.204008 23177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:10.205152 23177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:10.205190 23177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:10.205204 23177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:10.205242 23177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:10.205276 23177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:10.206750 23177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:10.206998 23177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:10.207218 23177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:10.211901 23182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:10.211937 23183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:10.211978 23177 server_base.cc:1034] running on GCE node
W20260926 06:29:10.211941 23185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:10.212350 23177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:10.212628 23177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:10.213783 23177 hybrid_clock.cc:648] HybridClock initialized: now 1790404150213770 us; error 25 us; skew 500 ppm
Sep 26 06:29:10 dist-test-slave-fkbn krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:10.216017 23177 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:10.216404 23177 webserver.cc:460] Webserver started at http://127.0.0.1:33489/ using document root <none> and password file <none>
I20260926 06:29:10.216641 23177 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:10.216706 23177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:10.216841 23177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:10.217938 23177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "330de8409f96405596cf5521a8b06e99"
format_stamp: "Formatted at 2026-09-26 06:29:10 on dist-test-slave-fkbn"
server_key: "14ad4ddc3ea974b36769ecb33c22dfc5"
server_key_iv: "1f443a3b2c179ecd61152899ede477d5"
server_key_version: "encryptionkey@0"
I20260926 06:29:10.218294 23177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "330de8409f96405596cf5521a8b06e99"
format_stamp: "Formatted at 2026-09-26 06:29:10 on dist-test-slave-fkbn"
server_key: "14ad4ddc3ea974b36769ecb33c22dfc5"
server_key_iv: "1f443a3b2c179ecd61152899ede477d5"
server_key_version: "encryptionkey@0"
I20260926 06:29:10.219529 23177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:29:10.220170 23192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:10.220323 23177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:29:10.220417 23177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "330de8409f96405596cf5521a8b06e99"
format_stamp: "Formatted at 2026-09-26 06:29:10 on dist-test-slave-fkbn"
server_key: "14ad4ddc3ea974b36769ecb33c22dfc5"
server_key_iv: "1f443a3b2c179ecd61152899ede477d5"
server_key_version: "encryptionkey@0"
I20260926 06:29:10.220492 23177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:10.234335 23177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:10.236025 23177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:10.236171 23177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:10.239933 23177 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41565
I20260926 06:29:10.239972 23244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41565 every 8 connection(s)
I20260926 06:29:10.240334 23177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:10.241372 23245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:10.243884 23245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99: Bootstrap starting.
I20260926 06:29:10.244551 23245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:10.244844 23245 log.cc:824] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:10.245558 23245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99: No bootstrap required, opened a new log
I20260926 06:29:10.247177 23245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330de8409f96405596cf5521a8b06e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260926 06:29:10.247306 23245 raft_consensus.cc:374] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:10.247350 23245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330de8409f96405596cf5521a8b06e99, State: Initialized, Role: FOLLOWER
I20260926 06:29:10.247509 23245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330de8409f96405596cf5521a8b06e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260926 06:29:10.247586 23245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:10.247634 23245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:10.247682 23245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:10.248175 23245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330de8409f96405596cf5521a8b06e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260926 06:29:10.248281 23245 leader_election.cc:302] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 330de8409f96405596cf5521a8b06e99; no voters:
I20260926 06:29:10.248447 23245 leader_election.cc:288] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:10.248539 23248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:10.248662 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 23177
I20260926 06:29:10.248698 23245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:10.248747 23248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [term 1 LEADER]: Becoming Leader. State: Replica: 330de8409f96405596cf5521a8b06e99, State: Running, Role: LEADER
I20260926 06:29:10.248787 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:10.248842 23248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330de8409f96405596cf5521a8b06e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } }
I20260926 06:29:10.249006 2207 external_mini_cluster.cc:1428] Setting key 3e8767f614835e994d43c6991608f5ef
I20260926 06:29:10.249895 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 330de8409f96405596cf5521a8b06e99. Latest consensus state: current_term: 1 leader_uuid: "330de8409f96405596cf5521a8b06e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330de8409f96405596cf5521a8b06e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } } }
I20260926 06:29:10.249996 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:10.250049 23249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "330de8409f96405596cf5521a8b06e99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330de8409f96405596cf5521a8b06e99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41565 } } }
I20260926 06:29:10.250356 23249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99 [sys.catalog]: This master's current role is: LEADER
W20260926 06:29:10.250662 23265 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:29:10.250723 23265 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:10.250778 23267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:10.250926 23267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:10.252272 23267 catalog_manager.cc:1347] Generated new cluster ID: e8d1cec34f39479a8559410429e94f8d
I20260926 06:29:10.252321 23267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:10.255709 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:10.249660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:50373 (local address 127.0.0.1:41565)
0926 06:29:10.249841 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:10.250109 (+ 268us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:10.250147 (+ 38us) server_negotiation.cc:407] Connection header received
0926 06:29:10.250461 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:10.250468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:10.250527 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:10.250909 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:10.251008 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:10.252354 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:10.253476 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:10.254116 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:10.254528 (+ 412us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:10.254920 (+ 392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:10.254923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:10.254924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:10.255145 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:10.255182 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:10.255198 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:10.255433 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:10.255476 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 06:29:10.255508 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260926 06:29:10.256574 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:10.259495 23267 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:10.259999 23267 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:10.263005 23267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 330de8409f96405596cf5521a8b06e99: Generated new TSK 0
I20260926 06:29:10.263156 23267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:10.264835 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:10.258167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:40111 (local address 127.0.0.1:41565)
0926 06:29:10.258198 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:10.258200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:10.261739 (+ 3539us) server_negotiation.cc:407] Connection header received
0926 06:29:10.261798 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:10.261799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:10.261810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:10.261868 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:10.262007 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:10.262736 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:10.263638 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:10.263879 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:10.263967 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:10.264614 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:10.264616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:10.264616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:10.264620 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:10.264634 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:10.264638 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:10.264691 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:10.264750 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 06:29:10.264761 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 06:29:10.266556 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 23177
W20260926 06:29:10.270711 23162 connection.cc:535] client connection to 127.0.0.1:41565 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41565): Connection reset by peer (error 104)
2026-09-26T06:29:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:10 dist-test-slave-fkbn krb5kdc[23277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:10 dist-test-slave-fkbn krb5kdc[23277](info): set up 2 sockets
Sep 26 06:29:10 dist-test-slave-fkbn krb5kdc[23277](info): commencing operation
krb5kdc: starting...
W20260926 06:29:11.297744 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:11 dist-test-slave-fkbn krb5kdc[23277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:11.412237 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:11.491817 23290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:11.492007 23290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:11.492028 23290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:11.493139 23290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:11.493180 23290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:11.493194 23290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:11.493239 23290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:11.493260 23290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:11.494781 23290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:11.495028 23290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:11.495247 23290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:11.499964 23295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:11.500001 23298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:11.500061 23290 server_base.cc:1034] running on GCE node
W20260926 06:29:11.499990 23296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:11.500391 23290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:11.500618 23290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:11.501773 23290 hybrid_clock.cc:648] HybridClock initialized: now 1790404151501762 us; error 36 us; skew 500 ppm
Sep 26 06:29:11 dist-test-slave-fkbn krb5kdc[23277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:11.504122 23290 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:11.504581 23290 webserver.cc:460] Webserver started at http://127.0.0.1:37767/ using document root <none> and password file <none>
I20260926 06:29:11.504798 23290 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:11.504860 23290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:11.505007 23290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:11.506083 23290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "779f03ab55a64e00bc0c12b6af609b59"
format_stamp: "Formatted at 2026-09-26 06:29:11 on dist-test-slave-fkbn"
server_key: "377e127d8254284dd959c6fa7e7bbb9e"
server_key_iv: "e568b4e86d5f40421804b199f9916f68"
server_key_version: "encryptionkey@0"
I20260926 06:29:11.506428 23290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "779f03ab55a64e00bc0c12b6af609b59"
format_stamp: "Formatted at 2026-09-26 06:29:11 on dist-test-slave-fkbn"
server_key: "377e127d8254284dd959c6fa7e7bbb9e"
server_key_iv: "e568b4e86d5f40421804b199f9916f68"
server_key_version: "encryptionkey@0"
I20260926 06:29:11.507733 23290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:29:11.508455 23305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:11.508608 23290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:29:11.508699 23290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "779f03ab55a64e00bc0c12b6af609b59"
format_stamp: "Formatted at 2026-09-26 06:29:11 on dist-test-slave-fkbn"
server_key: "377e127d8254284dd959c6fa7e7bbb9e"
server_key_iv: "e568b4e86d5f40421804b199f9916f68"
server_key_version: "encryptionkey@0"
I20260926 06:29:11.508759 23290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:11.511929 23290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:11.513722 23290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:11.513825 23290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:11.517649 23290 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33005
I20260926 06:29:11.517693 23357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33005 every 8 connection(s)
I20260926 06:29:11.518015 23290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:11.518940 23358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:11.521055 23358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59: Bootstrap starting.
I20260926 06:29:11.521684 23358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:11.521966 23358 log.cc:824] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:11.522637 23358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59: No bootstrap required, opened a new log
I20260926 06:29:11.524215 23358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779f03ab55a64e00bc0c12b6af609b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33005 } }
I20260926 06:29:11.524348 23358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:11.524399 23358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 779f03ab55a64e00bc0c12b6af609b59, State: Initialized, Role: FOLLOWER
I20260926 06:29:11.524573 23358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779f03ab55a64e00bc0c12b6af609b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33005 } }
I20260926 06:29:11.524652 23358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:11.524698 23358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:11.524748 23358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:11.525254 23358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779f03ab55a64e00bc0c12b6af609b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33005 } }
I20260926 06:29:11.525357 23358 leader_election.cc:302] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 779f03ab55a64e00bc0c12b6af609b59; no voters:
I20260926 06:29:11.525523 23358 leader_election.cc:288] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:11.525641 23361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:11.525792 23358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:11.525796 23361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [term 1 LEADER]: Becoming Leader. State: Replica: 779f03ab55a64e00bc0c12b6af609b59, State: Running, Role: LEADER
I20260926 06:29:11.525899 23361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779f03ab55a64e00bc0c12b6af609b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33005 } }
I20260926 06:29:11.526444 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 23290
I20260926 06:29:11.526589 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:11.526808 2207 external_mini_cluster.cc:1428] Setting key 1d543857a87e0267f373ecd0545191b4
I20260926 06:29:11.527307 23362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "779f03ab55a64e00bc0c12b6af609b59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779f03ab55a64e00bc0c12b6af609b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33005 } } }
I20260926 06:29:11.527273 23363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 779f03ab55a64e00bc0c12b6af609b59. Latest consensus state: current_term: 1 leader_uuid: "779f03ab55a64e00bc0c12b6af609b59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "779f03ab55a64e00bc0c12b6af609b59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33005 } } }
I20260926 06:29:11.527689 23363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:11.527679 23362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59 [sys.catalog]: This master's current role is: LEADER
W20260926 06:29:11.527887 23376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:29:11.527945 23376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:11.528719 23380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:11.528901 23380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:11.529212 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:11.527424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:56515 (local address 127.0.0.1:33005)
0926 06:29:11.527602 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:11.527610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:11.527642 (+ 32us) server_negotiation.cc:407] Connection header received
0926 06:29:11.527756 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:11.527763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:11.527794 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:11.528228 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:11.528631 (+ 403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:11.528634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:11.528634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:11.528638 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:11.528679 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:11.528695 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:11.528972 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:11.529001 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 06:29:11.529041 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260926 06:29:11.530076 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:11.532562 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:11.531615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:53671 (local address 127.0.0.1:33005)
0926 06:29:11.531650 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:11.531652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:11.531729 (+ 77us) server_negotiation.cc:407] Connection header received
0926 06:29:11.531791 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:11.531791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:11.531795 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:11.531835 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:11.532331 (+ 496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:11.532332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:11.532332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:11.532335 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:11.532347 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:11.532351 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:11.532394 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:11.532458 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 06:29:11.532468 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:29:11.533921 23380 catalog_manager.cc:1347] Generated new cluster ID: 67438d950553457d8006b566e43160a6
I20260926 06:29:11.533970 23380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:11.552654 23380 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:11.552862 23380 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:11.559640 23380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 779f03ab55a64e00bc0c12b6af609b59: Generated new TSK 0
I20260926 06:29:11.559839 23380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:11.566479 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 23290
2026-09-26T06:29:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:11 dist-test-slave-fkbn krb5kdc[23389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:11 dist-test-slave-fkbn krb5kdc[23389](info): set up 2 sockets
Sep 26 06:29:11 dist-test-slave-fkbn krb5kdc[23389](info): commencing operation
krb5kdc: starting...
W20260926 06:29:12.605523 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:12 dist-test-slave-fkbn krb5kdc[23389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:12.718047 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:12.796406 23402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:12.796600 23402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:12.796622 23402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:12.797734 23402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:12.797770 23402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:12.797783 23402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:12.797796 23402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:12.797808 23402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:12.799151 23402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43607
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:12.799363 23402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:12.799525 23402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:12.804208 23407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:12.804231 23408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:12.804278 23402 server_base.cc:1034] running on GCE node
W20260926 06:29:12.804217 23410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:12.804637 23402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:12.804910 23402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:12.806064 23402 hybrid_clock.cc:648] HybridClock initialized: now 1790404152806054 us; error 41 us; skew 500 ppm
Sep 26 06:29:12 dist-test-slave-fkbn krb5kdc[23389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:12.808374 23402 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:12.808779 23402 webserver.cc:460] Webserver started at http://127.0.0.1:37357/ using document root <none> and password file <none>
I20260926 06:29:12.809000 23402 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:12.809062 23402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:12.809207 23402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:12.810274 23402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "636ad6e033f64ca6a1ebedde3b906499"
format_stamp: "Formatted at 2026-09-26 06:29:12 on dist-test-slave-fkbn"
server_key: "382b6a8b1cd3e7352f16f203c38383f5"
server_key_iv: "f5fd18c46f6c90cad3061f5e664608eb"
server_key_version: "encryptionkey@0"
I20260926 06:29:12.810623 23402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "636ad6e033f64ca6a1ebedde3b906499"
format_stamp: "Formatted at 2026-09-26 06:29:12 on dist-test-slave-fkbn"
server_key: "382b6a8b1cd3e7352f16f203c38383f5"
server_key_iv: "f5fd18c46f6c90cad3061f5e664608eb"
server_key_version: "encryptionkey@0"
I20260926 06:29:12.811861 23402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:29:12.812554 23417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:12.812707 23402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:29:12.812801 23402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "636ad6e033f64ca6a1ebedde3b906499"
format_stamp: "Formatted at 2026-09-26 06:29:12 on dist-test-slave-fkbn"
server_key: "382b6a8b1cd3e7352f16f203c38383f5"
server_key_iv: "f5fd18c46f6c90cad3061f5e664608eb"
server_key_version: "encryptionkey@0"
I20260926 06:29:12.812867 23402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:12.850504 23402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:12.852782 23402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:12.852938 23402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:12.856734 23402 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43607
I20260926 06:29:12.856783 23469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43607 every 8 connection(s)
I20260926 06:29:12.857133 23402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:12.858110 23470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:12.860273 23470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499: Bootstrap starting.
I20260926 06:29:12.860827 23470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:12.861094 23470 log.cc:824] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:12.861835 23470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499: No bootstrap required, opened a new log
I20260926 06:29:12.862352 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 23402
I20260926 06:29:12.862479 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:12.862646 2207 external_mini_cluster.cc:1428] Setting key 120140a136f9cd1f053cd829e9a9a9df
I20260926 06:29:12.864073 23470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636ad6e033f64ca6a1ebedde3b906499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260926 06:29:12.864272 23470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:12.864307 23470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636ad6e033f64ca6a1ebedde3b906499, State: Initialized, Role: FOLLOWER
I20260926 06:29:12.864439 23470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636ad6e033f64ca6a1ebedde3b906499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260926 06:29:12.864523 23470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:12.864553 23470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:12.864605 23470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:12.865134 23470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636ad6e033f64ca6a1ebedde3b906499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260926 06:29:12.865237 23470 leader_election.cc:302] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 636ad6e033f64ca6a1ebedde3b906499; no voters:
I20260926 06:29:12.865388 23470 leader_election.cc:288] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:12.865454 23475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:12.865649 23475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [term 1 LEADER]: Becoming Leader. State: Replica: 636ad6e033f64ca6a1ebedde3b906499, State: Running, Role: LEADER
I20260926 06:29:12.865657 23470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:12.866771 23475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636ad6e033f64ca6a1ebedde3b906499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } }
I20260926 06:29:12.867326 23476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "636ad6e033f64ca6a1ebedde3b906499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636ad6e033f64ca6a1ebedde3b906499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } } }
I20260926 06:29:12.867952 23476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:12.867444 23477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 636ad6e033f64ca6a1ebedde3b906499. Latest consensus state: current_term: 1 leader_uuid: "636ad6e033f64ca6a1ebedde3b906499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636ad6e033f64ca6a1ebedde3b906499" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43607 } } }
W20260926 06:29:12.867704 23489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 06:29:12.868163 23477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499 [sys.catalog]: This master's current role is: LEADER
W20260926 06:29:12.868175 23489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:12.868314 23492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:12.868482 23492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:12.869910 23492 catalog_manager.cc:1347] Generated new cluster ID: b37eac53abfa4ee99e6dbd1f7e0f3477
I20260926 06:29:12.869963 23492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 06:29:12 dist-test-slave-fkbn krb5kdc[23389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 06:29:12.883841 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:12.863135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:37997 (local address 127.0.0.1:43607)
0926 06:29:12.863333 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:12.863343 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:12.863376 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:29:12.864265 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:12.864274 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:12.864340 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:12.864480 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:12.865128 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:12.866355 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:12.867489 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:12.868071 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:12.868172 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:12.880202 (+ 12030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:12.880207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:12.880208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:12.880222 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:12.882142 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:12.882572 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:12.882576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:12.882576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:12.882685 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:12.882903 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:12.882904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:12.882904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:12.883213 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:12.883377 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:12.883499 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 06:29:12.883604 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
I20260926 06:29:12.884734 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:12.888794 23492 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:12.889247 23492 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:12.893134 23492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 636ad6e033f64ca6a1ebedde3b906499: Generated new TSK 0
I20260926 06:29:12.893303 23492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:12.893956 23474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:12.886244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:38673 (local address 127.0.0.1:43607)
0926 06:29:12.886278 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:12.886280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:12.886512 (+ 232us) server_negotiation.cc:407] Connection header received
0926 06:29:12.886603 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:12.886604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:12.886614 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:12.886641 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:12.887303 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:12.888323 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:12.889364 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:12.889697 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:12.889738 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:12.891126 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:12.891127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:12.891128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:12.891134 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:12.892581 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:12.893016 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:12.893017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:12.893017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:12.893082 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:12.893359 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:12.893360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:12.893360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:12.893492 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:12.893554 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:12.893799 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 06:29:12.893861 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:29:12.895654 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 23402
W20260926 06:29:12.900118 23387 connection.cc:535] client connection to 127.0.0.1:43607 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43607): Connection reset by peer (error 104)
2026-09-26T06:29:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:12 dist-test-slave-fkbn krb5kdc[23501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:12 dist-test-slave-fkbn krb5kdc[23501](info): set up 2 sockets
Sep 26 06:29:12 dist-test-slave-fkbn krb5kdc[23501](info): commencing operation
krb5kdc: starting...
W20260926 06:29:13.921710 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:14 dist-test-slave-fkbn krb5kdc[23501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:14.040011 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:14.119148 23514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:14.119381 23514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:14.119418 23514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:14.120541 23514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:14.120602 23514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:14.120631 23514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:14.120659 23514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:14.120682 23514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:14.122174 23514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:14.122442 23514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:14.122666 23514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:14.127425 23519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:14.127434 23522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:14.127586 23514 server_base.cc:1034] running on GCE node
W20260926 06:29:14.127480 23520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:14.127892 23514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:14.128165 23514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:14.129311 23514 hybrid_clock.cc:648] HybridClock initialized: now 1790404154129306 us; error 45 us; skew 500 ppm
Sep 26 06:29:14 dist-test-slave-fkbn krb5kdc[23501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:14.137799 23514 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:14.138244 23514 webserver.cc:460] Webserver started at http://127.0.0.1:33853/ using document root <none> and password file <none>
I20260926 06:29:14.138494 23514 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:14.138562 23514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:14.138708 23514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:14.139848 23514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "43393c4cf76e4f94ac199ed11de9e9f7"
format_stamp: "Formatted at 2026-09-26 06:29:14 on dist-test-slave-fkbn"
server_key: "de48716ecb07431a77d80a600c08ff04"
server_key_iv: "2358c18c05e100ebc86558216608d98f"
server_key_version: "encryptionkey@0"
I20260926 06:29:14.140242 23514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "43393c4cf76e4f94ac199ed11de9e9f7"
format_stamp: "Formatted at 2026-09-26 06:29:14 on dist-test-slave-fkbn"
server_key: "de48716ecb07431a77d80a600c08ff04"
server_key_iv: "2358c18c05e100ebc86558216608d98f"
server_key_version: "encryptionkey@0"
I20260926 06:29:14.141669 23514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:29:14.142454 23529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:14.142666 23514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:29:14.142796 23514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "43393c4cf76e4f94ac199ed11de9e9f7"
format_stamp: "Formatted at 2026-09-26 06:29:14 on dist-test-slave-fkbn"
server_key: "de48716ecb07431a77d80a600c08ff04"
server_key_iv: "2358c18c05e100ebc86558216608d98f"
server_key_version: "encryptionkey@0"
I20260926 06:29:14.142872 23514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:14.162115 23514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:14.164641 23514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:14.164811 23514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:14.168931 23514 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33043
I20260926 06:29:14.168998 23581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33043 every 8 connection(s)
I20260926 06:29:14.169370 23514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:14.170245 23582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:14.172571 23582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7: Bootstrap starting.
I20260926 06:29:14.173135 23582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:14.173416 23582 log.cc:824] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:14.174126 23582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7: No bootstrap required, opened a new log
I20260926 06:29:14.174575 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 23514
I20260926 06:29:14.174691 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:14.174875 2207 external_mini_cluster.cc:1428] Setting key f4625b44e12d69305df2204a2622d52e
I20260926 06:29:14.176308 23582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43393c4cf76e4f94ac199ed11de9e9f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33043 } }
I20260926 06:29:14.176522 23582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:14.176558 23582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43393c4cf76e4f94ac199ed11de9e9f7, State: Initialized, Role: FOLLOWER
I20260926 06:29:14.176679 23582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43393c4cf76e4f94ac199ed11de9e9f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33043 } }
I20260926 06:29:14.176764 23582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:14.176793 23582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:14.176846 23582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:14.177510 23582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43393c4cf76e4f94ac199ed11de9e9f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33043 } }
I20260926 06:29:14.177651 23582 leader_election.cc:302] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43393c4cf76e4f94ac199ed11de9e9f7; no voters:
I20260926 06:29:14.177793 23582 leader_election.cc:288] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:14.177845 23587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:14.178014 23582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:14.178025 23587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [term 1 LEADER]: Becoming Leader. State: Replica: 43393c4cf76e4f94ac199ed11de9e9f7, State: Running, Role: LEADER
I20260926 06:29:14.178133 23587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43393c4cf76e4f94ac199ed11de9e9f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33043 } }
I20260926 06:29:14.178499 23589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43393c4cf76e4f94ac199ed11de9e9f7. Latest consensus state: current_term: 1 leader_uuid: "43393c4cf76e4f94ac199ed11de9e9f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43393c4cf76e4f94ac199ed11de9e9f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33043 } } }
I20260926 06:29:14.178506 23588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43393c4cf76e4f94ac199ed11de9e9f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43393c4cf76e4f94ac199ed11de9e9f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33043 } } }
I20260926 06:29:14.178613 23589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:14.178617 23588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:14.178859 23596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:14.179355 23596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:14.181309 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:14.175389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:36663 (local address 127.0.0.1:33043)
0926 06:29:14.175601 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:14.175610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:14.175644 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:29:14.176505 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:14.176513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:14.176579 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:14.176722 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:14.176949 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:14.178326 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:14.179481 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:14.179899 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:14.179996 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:14.180666 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:14.180670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:14.180670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:14.180836 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:14.180864 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:14.180874 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:14.180960 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:14.181060 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 06:29:14.181103 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260926 06:29:14.182293 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:14.182894 23596 catalog_manager.cc:1347] Generated new cluster ID: b080436e351f47449741e8011592bc36
I20260926 06:29:14.182948 23596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:14.193025 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:14.185943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:53377 (local address 127.0.0.1:33043)
0926 06:29:14.185989 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:14.185990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:14.189723 (+ 3733us) server_negotiation.cc:407] Connection header received
0926 06:29:14.189774 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:14.189775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:14.189785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:14.189842 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:14.189979 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:14.190714 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:14.191679 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:14.192003 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:14.192073 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:14.192768 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:14.192769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:14.192769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:14.192774 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:14.192790 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:14.192794 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:14.192849 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:14.192925 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 06:29:14.192935 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 06:29:14.200635 23596 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:14.201167 23596 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:14.211073 23596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43393c4cf76e4f94ac199ed11de9e9f7: Generated new TSK 0
I20260926 06:29:14.211261 23596 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:14.215423 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 23514
W20260926 06:29:14.219735 23499 connection.cc:535] client connection to 127.0.0.1:33043 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33043): Connection reset by peer (error 104)
2026-09-26T06:29:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:14 dist-test-slave-fkbn krb5kdc[23613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:14 dist-test-slave-fkbn krb5kdc[23613](info): set up 2 sockets
Sep 26 06:29:14 dist-test-slave-fkbn krb5kdc[23613](info): commencing operation
krb5kdc: starting...
W20260926 06:29:15.241627 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:15 dist-test-slave-fkbn krb5kdc[23613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 06:29:15.356552 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:15.436120 23626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:15.436328 23626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:15.436354 23626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:15.437623 23626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:15.437666 23626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:15.437682 23626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:15.437731 23626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:15.437760 23626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:15.439227 23626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:15.439472 23626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:15.439692 23626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:15.444726 23631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:15.444743 23632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:15.444726 23634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:15.444746 23626 server_base.cc:1034] running on GCE node
I20260926 06:29:15.445222 23626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:15.445500 23626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:15.446655 23626 hybrid_clock.cc:648] HybridClock initialized: now 1790404155446656 us; error 44 us; skew 500 ppm
Sep 26 06:29:15 dist-test-slave-fkbn krb5kdc[23613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:15.448990 23626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 06:29:15.449380 23626 webserver.cc:460] Webserver started at http://127.0.0.1:35937/ using document root <none> and password file <none>
I20260926 06:29:15.449632 23626 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:15.449697 23626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:15.449834 23626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:15.450826 23626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "42992377de9d45908203dbe8e535de95"
format_stamp: "Formatted at 2026-09-26 06:29:15 on dist-test-slave-fkbn"
server_key: "cbc2c64a138d9afad8e05593bc6aa2a3"
server_key_iv: "f747de58aa8b69e5996c88b9c539c710"
server_key_version: "encryptionkey@0"
I20260926 06:29:15.451189 23626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "42992377de9d45908203dbe8e535de95"
format_stamp: "Formatted at 2026-09-26 06:29:15 on dist-test-slave-fkbn"
server_key: "cbc2c64a138d9afad8e05593bc6aa2a3"
server_key_iv: "f747de58aa8b69e5996c88b9c539c710"
server_key_version: "encryptionkey@0"
I20260926 06:29:15.452430 23626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:29:15.453110 23641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:15.453269 23626 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:29:15.453358 23626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "42992377de9d45908203dbe8e535de95"
format_stamp: "Formatted at 2026-09-26 06:29:15 on dist-test-slave-fkbn"
server_key: "cbc2c64a138d9afad8e05593bc6aa2a3"
server_key_iv: "f747de58aa8b69e5996c88b9c539c710"
server_key_version: "encryptionkey@0"
I20260926 06:29:15.453436 23626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:15.456532 23626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:15.458317 23626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:15.458464 23626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:15.462314 23626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32939
I20260926 06:29:15.462347 23693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32939 every 8 connection(s)
I20260926 06:29:15.462644 23626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:15.463517 23694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:15.465664 23694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95: Bootstrap starting.
I20260926 06:29:15.466195 23694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:15.466475 23694 log.cc:824] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:15.467103 23694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95: No bootstrap required, opened a new log
I20260926 06:29:15.468617 23694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42992377de9d45908203dbe8e535de95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260926 06:29:15.468739 23694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:15.468783 23694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42992377de9d45908203dbe8e535de95, State: Initialized, Role: FOLLOWER
I20260926 06:29:15.468957 23694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42992377de9d45908203dbe8e535de95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260926 06:29:15.469031 23694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:15.469075 23694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:15.469123 23694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:15.469667 23694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42992377de9d45908203dbe8e535de95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260926 06:29:15.469764 23694 leader_election.cc:302] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42992377de9d45908203dbe8e535de95; no voters:
I20260926 06:29:15.469936 23694 leader_election.cc:288] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:15.469951 23697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:15.470089 23697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [term 1 LEADER]: Becoming Leader. State: Replica: 42992377de9d45908203dbe8e535de95, State: Running, Role: LEADER
I20260926 06:29:15.470170 23694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:15.470216 23697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42992377de9d45908203dbe8e535de95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260926 06:29:15.470625 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 23626
I20260926 06:29:15.470743 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:15.470896 2207 external_mini_cluster.cc:1428] Setting key e1e8ec6039a7b0d0f2ca7fb996408889
I20260926 06:29:15.471336 23698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42992377de9d45908203dbe8e535de95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42992377de9d45908203dbe8e535de95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } } }
I20260926 06:29:15.471422 23698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:15.471549 23699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42992377de9d45908203dbe8e535de95. Latest consensus state: current_term: 1 leader_uuid: "42992377de9d45908203dbe8e535de95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42992377de9d45908203dbe8e535de95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } } }
I20260926 06:29:15.471689 23699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:15.471819 23712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:15.472297 23712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:15.473770 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:15.471317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:47221 (local address 127.0.0.1:32939)
0926 06:29:15.471470 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:15.471479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:15.471878 (+ 399us) server_negotiation.cc:407] Connection header received
0926 06:29:15.472109 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:15.472117 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:15.472148 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:15.472254 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:15.473251 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:15.473255 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:15.473255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:15.473259 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:15.473305 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:15.473317 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:15.473396 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:15.473539 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 06:29:15.473607 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260926 06:29:15.474321 23712 catalog_manager.cc:1347] Generated new cluster ID: 34756a6f3fe243e2ae937ab0e7e1b406
I20260926 06:29:15.474373 23712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:15.474643 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:15.478785 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:15.477630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.61:42321 (local address 127.0.0.1:32939)
0926 06:29:15.477666 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:15.477669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:15.477833 (+ 164us) server_negotiation.cc:407] Connection header received
0926 06:29:15.477910 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:15.477910 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:15.477918 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:15.477993 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:15.478545 (+ 552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:15.478546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:15.478547 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:15.478551 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:15.478563 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:15.478567 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:15.478622 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:15.478678 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 06:29:15.478693 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:29:15.490871 23712 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:15.491024 23712 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:15.499868 23712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42992377de9d45908203dbe8e535de95: Generated new TSK 0
I20260926 06:29:15.500034 23712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:15.503432 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 23626
2026-09-26T06:29:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:15 dist-test-slave-fkbn krb5kdc[23725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:15 dist-test-slave-fkbn krb5kdc[23725](info): set up 2 sockets
Sep 26 06:29:15 dist-test-slave-fkbn krb5kdc[23725](info): commencing operation
krb5kdc: starting...
W20260926 06:29:16.621738 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:16 dist-test-slave-fkbn krb5kdc[23725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:29:16.741196 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:44111
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:44711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:44111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:16.822798 23738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:16.823025 23738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:16.823060 23738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:16.824195 23738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:16.824260 23738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:16.824291 23738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:16.824318 23738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:16.824348 23738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:16.825809 23738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:44711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:44111
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:44111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:16.826074 23738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:16.826292 23738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:16.830901 23743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:16.830938 23744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:16.830910 23746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:16.831007 23738 server_base.cc:1034] running on GCE node
I20260926 06:29:16.831388 23738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:16.831672 23738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:16.832849 23738 hybrid_clock.cc:648] HybridClock initialized: now 1790404156832835 us; error 41 us; skew 500 ppm
Sep 26 06:29:16 dist-test-slave-fkbn krb5kdc[23725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404156, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:16.835320 23738 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:29:16.835718 23738 webserver.cc:460] Webserver started at http://127.2.39.254:37589/ using document root <none> and password file <none>
I20260926 06:29:16.835889 23738 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:16.835935 23738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:16.836031 23738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:16.836953 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "e29d006dd8124ead994db863f2633b52"
format_stamp: "Formatted at 2026-09-26 06:29:16 on dist-test-slave-fkbn"
server_key: "7ee4c0e334bf574b8d5f53c81f5dd139"
server_key_iv: "ff54605805bcae229b1c8d3010c4d20d"
server_key_version: "encryptionkey@0"
I20260926 06:29:16.837237 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "e29d006dd8124ead994db863f2633b52"
format_stamp: "Formatted at 2026-09-26 06:29:16 on dist-test-slave-fkbn"
server_key: "7ee4c0e334bf574b8d5f53c81f5dd139"
server_key_iv: "ff54605805bcae229b1c8d3010c4d20d"
server_key_version: "encryptionkey@0"
I20260926 06:29:16.838577 23738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:29:16.839241 23753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:16.839399 23738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:29:16.839491 23738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "e29d006dd8124ead994db863f2633b52"
format_stamp: "Formatted at 2026-09-26 06:29:16 on dist-test-slave-fkbn"
server_key: "7ee4c0e334bf574b8d5f53c81f5dd139"
server_key_iv: "ff54605805bcae229b1c8d3010c4d20d"
server_key_version: "encryptionkey@0"
I20260926 06:29:16.839566 23738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:16.858012 23738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:16.860215 23738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:16.860376 23738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:16.864511 23738 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:44111
I20260926 06:29:16.864548 23805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:44111 every 8 connection(s)
I20260926 06:29:16.864914 23738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:16.865656 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 23738
I20260926 06:29:16.865780 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:16.865844 23806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:16.865996 2207 external_mini_cluster.cc:1428] Setting key 54ceeac91e957d61a77579e23577fb13
I20260926 06:29:16.869148 23806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52: Bootstrap starting.
I20260926 06:29:16.869871 23806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:16.870157 23806 log.cc:824] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:16.870796 23806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52: No bootstrap required, opened a new log
I20260926 06:29:16.872545 23806 raft_consensus.cc:348] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d006dd8124ead994db863f2633b52" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 44111 } }
Sep 26 06:29:16 dist-test-slave-fkbn krb5kdc[23725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:29:16.872695 23806 raft_consensus.cc:374] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:16.872747 23806 raft_consensus.cc:729] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e29d006dd8124ead994db863f2633b52, State: Initialized, Role: FOLLOWER
I20260926 06:29:16.872886 23806 consensus_queue.cc:260] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d006dd8124ead994db863f2633b52" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 44111 } }
I20260926 06:29:16.872982 23806 raft_consensus.cc:388] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:16.873026 23806 raft_consensus.cc:482] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:16.873073 23806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:16.873857 23806 raft_consensus.cc:504] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d006dd8124ead994db863f2633b52" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 44111 } }
I20260926 06:29:16.873976 23806 leader_election.cc:302] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e29d006dd8124ead994db863f2633b52; no voters:
I20260926 06:29:16.874145 23806 leader_election.cc:288] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:16.874195 23811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:16.874374 23811 raft_consensus.cc:686] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [term 1 LEADER]: Becoming Leader. State: Replica: e29d006dd8124ead994db863f2633b52, State: Running, Role: LEADER
I20260926 06:29:16.874423 23806 sys_catalog.cc:564] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:16.874558 23811 consensus_queue.cc:237] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d006dd8124ead994db863f2633b52" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 44111 } }
I20260926 06:29:16.874898 23811 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e29d006dd8124ead994db863f2633b52. Latest consensus state: current_term: 1 leader_uuid: "e29d006dd8124ead994db863f2633b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d006dd8124ead994db863f2633b52" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 44111 } } }
I20260926 06:29:16.875147 23811 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:16.875049 23812 sys_catalog.cc:455] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e29d006dd8124ead994db863f2633b52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e29d006dd8124ead994db863f2633b52" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 44111 } } }
I20260926 06:29:16.875540 23812 sys_catalog.cc:458] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:16.875551 23820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:16.876226 23820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:16.877204 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:16.866476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39877 (local address 127.2.39.254:44111)
0926 06:29:16.866668 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:16.866678 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:16.866711 (+ 33us) server_negotiation.cc:407] Connection header received
0926 06:29:16.867386 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:16.867395 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:16.867459 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:16.867611 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:16.868186 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:16.869562 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:16.870402 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:16.870804 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:16.870913 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:16.873241 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:16.873246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:16.873246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:16.873257 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:16.875324 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:16.875995 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:16.875998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:16.875999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:16.876146 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:16.876379 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:16.876380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:16.876381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:16.876607 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:16.876777 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:16.876904 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 06:29:16.877015 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260926 06:29:16.878100 23820 catalog_manager.cc:1347] Generated new cluster ID: bd6a60dc246d4a899f17011703047a56
I20260926 06:29:16.878284 23820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:16.878378 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:16.885836 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:16.879755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35739 (local address 127.2.39.254:44111)
0926 06:29:16.879790 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:16.879792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:16.879902 (+ 110us) server_negotiation.cc:407] Connection header received
0926 06:29:16.879978 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:16.879979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:16.879993 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:16.880038 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:16.880543 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:16.881017 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:16.881774 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:16.882002 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:16.882090 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:16.883257 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:16.883258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:16.883259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:16.883266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:16.884637 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:16.885000 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:16.885001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:16.885002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:16.885060 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:16.885281 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:16.885282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:16.885282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:16.885392 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:16.885507 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:16.885661 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 06:29:16.885722 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:29:16.893899 23820 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:16.894403 23820 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:16.904840 23820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e29d006dd8124ead994db863f2633b52: Generated new TSK 0
I20260926 06:29:16.905058 23820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:16.909850 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 23738
W20260926 06:29:16.914438 23723 connection.cc:535] client connection to 127.2.39.254:44111 recv error: Network error: failed to read from TLS socket (remote: 127.2.39.254:44111): Connection reset by peer (error 104)
2026-09-26T06:29:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:16 dist-test-slave-fkbn krb5kdc[23837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:16 dist-test-slave-fkbn krb5kdc[23837](info): set up 2 sockets
Sep 26 06:29:16 dist-test-slave-fkbn krb5kdc[23837](info): commencing operation
krb5kdc: starting...
W20260926 06:29:17.993799 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:18 dist-test-slave-fkbn krb5kdc[23837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:29:18.112782 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:37141
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:37141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:18.195374 23850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:18.195588 23850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:18.195612 23850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:18.196805 23850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:18.196851 23850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:18.196866 23850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:18.196878 23850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:18.196933 23850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:18.198678 23850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:37141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:37141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:18.198953 23850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:18.199190 23850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:18.204290 23855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:18.204308 23856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:18.204365 23850 server_base.cc:1034] running on GCE node
W20260926 06:29:18.204293 23858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:18.204746 23850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:18.205046 23850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:18.206189 23850 hybrid_clock.cc:648] HybridClock initialized: now 1790404158206169 us; error 40 us; skew 500 ppm
Sep 26 06:29:18 dist-test-slave-fkbn krb5kdc[23837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404158, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:18.208655 23850 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:29:18.209085 23850 webserver.cc:460] Webserver started at http://127.2.39.254:35053/ using document root <none> and password file <none>
I20260926 06:29:18.209323 23850 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:18.209367 23850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:18.209534 23850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:18.210639 23850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3"
format_stamp: "Formatted at 2026-09-26 06:29:18 on dist-test-slave-fkbn"
server_key: "77f90029c368abbc4996e4b9d1fa4d23"
server_key_iv: "d9b2995baba2ef6a364b9a3a40cb69eb"
server_key_version: "encryptionkey@0"
I20260926 06:29:18.210989 23850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3"
format_stamp: "Formatted at 2026-09-26 06:29:18 on dist-test-slave-fkbn"
server_key: "77f90029c368abbc4996e4b9d1fa4d23"
server_key_iv: "d9b2995baba2ef6a364b9a3a40cb69eb"
server_key_version: "encryptionkey@0"
I20260926 06:29:18.212316 23850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:29:18.213073 23865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:18.213250 23850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 06:29:18.213351 23850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3"
format_stamp: "Formatted at 2026-09-26 06:29:18 on dist-test-slave-fkbn"
server_key: "77f90029c368abbc4996e4b9d1fa4d23"
server_key_iv: "d9b2995baba2ef6a364b9a3a40cb69eb"
server_key_version: "encryptionkey@0"
I20260926 06:29:18.213456 23850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:18.241937 23850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:18.243944 23850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:18.244102 23850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:18.248101 23850 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:37141
I20260926 06:29:18.248157 23917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:37141 every 8 connection(s)
I20260926 06:29:18.248538 23850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:18.249377 23918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:18.251678 23918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3: Bootstrap starting.
I20260926 06:29:18.252259 23918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:18.252542 23918 log.cc:824] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:18.253253 23918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3: No bootstrap required, opened a new log
I20260926 06:29:18.254855 23918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37141 } }
I20260926 06:29:18.255025 23918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:18.255087 23918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7ae0a5cbd4419aa03c90cfc7539da3, State: Initialized, Role: FOLLOWER
I20260926 06:29:18.255276 23918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37141 } }
I20260926 06:29:18.255359 23918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:18.255409 23918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:18.255461 23918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:18.256273 23918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37141 } }
I20260926 06:29:18.256383 23918 leader_election.cc:302] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c7ae0a5cbd4419aa03c90cfc7539da3; no voters:
I20260926 06:29:18.256565 23918 leader_election.cc:288] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:18.256597 23921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:18.256767 23921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c7ae0a5cbd4419aa03c90cfc7539da3, State: Running, Role: LEADER
I20260926 06:29:18.256820 23918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:18.256901 23921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37141 } }
I20260926 06:29:18.257275 23922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37141 } } }
I20260926 06:29:18.257355 23922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:18.257485 23923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c7ae0a5cbd4419aa03c90cfc7539da3. Latest consensus state: current_term: 1 leader_uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7ae0a5cbd4419aa03c90cfc7539da3" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 37141 } } }
I20260926 06:29:18.257652 23923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:18.257961 23930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:18.258248 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 23850
I20260926 06:29:18.258365 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:18.258477 23930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:18.258564 2207 external_mini_cluster.cc:1428] Setting key 5dd32a03e942819663bcce93fbd06709
I20260926 06:29:18.260136 23930 catalog_manager.cc:1347] Generated new cluster ID: 7f162c5107344a7e92104f66a4832c97
I20260926 06:29:18.260188 23930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:18.265309 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:18.259024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36625 (local address 127.2.39.254:37141)
0926 06:29:18.259575 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:18.259585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:18.259614 (+ 29us) server_negotiation.cc:407] Connection header received
0926 06:29:18.259866 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:18.259873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:18.259931 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:18.260089 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:18.261720 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:18.262620 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:18.263450 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:18.263852 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:18.263985 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:18.264677 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:18.264681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:18.264681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:18.264856 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:18.264885 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:18.264896 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:18.264994 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:18.265072 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 06:29:18.265101 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":72,"threads_started":1}
I20260926 06:29:18.266403 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:18.266620 23930 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:18.267236 23930 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:18.275308 23930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c7ae0a5cbd4419aa03c90cfc7539da3: Generated new TSK 0
I20260926 06:29:18.275487 23930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:18.276793 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:18.269859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36305 (local address 127.2.39.254:37141)
0926 06:29:18.269909 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:18.269910 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:18.273749 (+ 3839us) server_negotiation.cc:407] Connection header received
0926 06:29:18.273813 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:18.273814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:18.273826 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:18.273908 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:18.274070 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:18.274579 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:18.275558 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:18.275790 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:18.275824 (+ 34us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:18.276568 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:18.276569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:18.276570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:18.276574 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:18.276587 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:18.276591 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:18.276644 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:18.276705 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 06:29:18.276717 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:29:18.278610 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 23850
W20260926 06:29:18.282452 23835 connection.cc:535] client connection to 127.2.39.254:37141 recv error: Network error: failed to read from TLS socket (remote: 127.2.39.254:37141): Connection reset by peer (error 104)
2026-09-26T06:29:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:18 dist-test-slave-fkbn krb5kdc[23949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:18 dist-test-slave-fkbn krb5kdc[23949](info): set up 2 sockets
Sep 26 06:29:18 dist-test-slave-fkbn krb5kdc[23949](info): commencing operation
krb5kdc: starting...
W20260926 06:29:19.329799 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:19 dist-test-slave-fkbn krb5kdc[23949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:29:19.449359 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:45503
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:19.529677 23962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:19.529940 23962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:19.529980 23962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:19.531133 23962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:19.531206 23962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:19.531230 23962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:19.531260 23962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:19.531287 23962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:19.532835 23962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:45503
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:19.533128 23962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:19.533366 23962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:19.538381 23968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:19.538388 23970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:19.538446 23967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:19.538451 23962 server_base.cc:1034] running on GCE node
I20260926 06:29:19.538831 23962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:19.539127 23962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:19.540292 23962 hybrid_clock.cc:648] HybridClock initialized: now 1790404159540271 us; error 38 us; skew 500 ppm
Sep 26 06:29:19 dist-test-slave-fkbn krb5kdc[23949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404159, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:19.548575 23962 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:29:19.549084 23962 webserver.cc:460] Webserver started at http://127.2.39.254:35429/ using document root <none> and password file <none>
I20260926 06:29:19.549341 23962 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:19.549388 23962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:19.549530 23962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:19.550690 23962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "37723d22475c4b54828d825f116e26bd"
format_stamp: "Formatted at 2026-09-26 06:29:19 on dist-test-slave-fkbn"
server_key: "7f42ac4b5d2ec2eae13a85b0b2254e23"
server_key_iv: "40110def8e81c8b62188983c8527a7e7"
server_key_version: "encryptionkey@0"
I20260926 06:29:19.551056 23962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "37723d22475c4b54828d825f116e26bd"
format_stamp: "Formatted at 2026-09-26 06:29:19 on dist-test-slave-fkbn"
server_key: "7f42ac4b5d2ec2eae13a85b0b2254e23"
server_key_iv: "40110def8e81c8b62188983c8527a7e7"
server_key_version: "encryptionkey@0"
I20260926 06:29:19.552393 23962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:29:19.553171 23977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:19.553365 23962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:29:19.553498 23962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "37723d22475c4b54828d825f116e26bd"
format_stamp: "Formatted at 2026-09-26 06:29:19 on dist-test-slave-fkbn"
server_key: "7f42ac4b5d2ec2eae13a85b0b2254e23"
server_key_iv: "40110def8e81c8b62188983c8527a7e7"
server_key_version: "encryptionkey@0"
I20260926 06:29:19.553576 23962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:19.556905 23962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:19.558888 23962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:19.559043 23962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:19.563144 23962 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:45503
I20260926 06:29:19.563180 24029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:45503 every 8 connection(s)
I20260926 06:29:19.563563 23962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:19.564553 24030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:19.566838 24030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd: Bootstrap starting.
I20260926 06:29:19.567354 24030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:19.567590 24030 log.cc:824] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:19.568215 24030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd: No bootstrap required, opened a new log
I20260926 06:29:19.569828 24030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37723d22475c4b54828d825f116e26bd" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45503 } }
I20260926 06:29:19.569964 24030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:19.569988 24030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37723d22475c4b54828d825f116e26bd, State: Initialized, Role: FOLLOWER
I20260926 06:29:19.570118 24030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37723d22475c4b54828d825f116e26bd" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45503 } }
I20260926 06:29:19.570199 24030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:19.570221 24030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:19.570250 24030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:19.571031 24030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37723d22475c4b54828d825f116e26bd" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45503 } }
I20260926 06:29:19.571115 24030 leader_election.cc:302] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37723d22475c4b54828d825f116e26bd; no voters:
I20260926 06:29:19.571228 24030 leader_election.cc:288] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:19.571292 24033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:19.571471 24033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [term 1 LEADER]: Becoming Leader. State: Replica: 37723d22475c4b54828d825f116e26bd, State: Running, Role: LEADER
I20260926 06:29:19.571504 24030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:19.571557 24033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37723d22475c4b54828d825f116e26bd" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45503 } }
I20260926 06:29:19.571930 24034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37723d22475c4b54828d825f116e26bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37723d22475c4b54828d825f116e26bd" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45503 } } }
I20260926 06:29:19.572026 24035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37723d22475c4b54828d825f116e26bd. Latest consensus state: current_term: 1 leader_uuid: "37723d22475c4b54828d825f116e26bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37723d22475c4b54828d825f116e26bd" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 45503 } } }
I20260926 06:29:19.572124 24035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:19.572363 24040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:19.572031 24034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:19.572928 24040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:19.573769 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 23962
I20260926 06:29:19.573874 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:19.574103 2207 external_mini_cluster.cc:1428] Setting key 556886617704e8c0cb10af9a980f6409
I20260926 06:29:19.574589 24040 catalog_manager.cc:1347] Generated new cluster ID: 3b4371d5f54e447dad1565eb45702c09
I20260926 06:29:19.574640 24040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:19.583218 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:19.574644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39187 (local address 127.2.39.254:45503)
0926 06:29:19.581709 (+ 7065us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:19.581716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:19.581748 (+ 32us) server_negotiation.cc:407] Connection header received
0926 06:29:19.581850 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:19.581856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:19.581879 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:19.582078 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:19.582760 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:19.582763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:19.582764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:19.582768 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:19.582821 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:19.582832 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:19.582906 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:19.582973 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 06:29:19.583019 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7019,"thread_start_us":73,"threads_started":1}
I20260926 06:29:19.584059 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:19.586655 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:19.585422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60055 (local address 127.2.39.254:45503)
0926 06:29:19.585462 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:19.585464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:19.585705 (+ 241us) server_negotiation.cc:407] Connection header received
0926 06:29:19.585784 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:19.585785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:19.585790 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:19.585843 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:19.586435 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:19.586436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:19.586437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:19.586440 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:19.586454 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:19.586458 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:19.586508 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:19.586561 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 06:29:19.586575 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:29:19.594995 24040 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:19.595182 24040 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:19.602345 24040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37723d22475c4b54828d825f116e26bd: Generated new TSK 0
I20260926 06:29:19.602594 24040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:19.604455 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 23962
2026-09-26T06:29:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:19 dist-test-slave-fkbn krb5kdc[24061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:19 dist-test-slave-fkbn krb5kdc[24061](info): set up 2 sockets
Sep 26 06:29:19 dist-test-slave-fkbn krb5kdc[24061](info): commencing operation
krb5kdc: starting...
W20260926 06:29:20.633744 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:20 dist-test-slave-fkbn krb5kdc[24061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:29:20.756853 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:34355
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:33511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:34355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:20.837244 24074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:20.837450 24074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:20.837473 24074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:20.838645 24074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:20.838688 24074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:20.838702 24074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:20.838738 24074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:20.838766 24074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:20.840220 24074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:33511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:34355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:34355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:20.840471 24074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:20.840696 24074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:20.845467 24082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:20.845544 24080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:20.845495 24079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:20.845693 24074 server_base.cc:1034] running on GCE node
I20260926 06:29:20.845906 24074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:20.846158 24074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:20.847297 24074 hybrid_clock.cc:648] HybridClock initialized: now 1790404160847286 us; error 29 us; skew 500 ppm
Sep 26 06:29:20 dist-test-slave-fkbn krb5kdc[24061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404160, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:20.849738 24074 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:29:20.850159 24074 webserver.cc:460] Webserver started at http://127.2.39.254:39061/ using document root <none> and password file <none>
I20260926 06:29:20.850384 24074 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:20.850440 24074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:20.850593 24074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:20.851686 24074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "0bf08391de1a496fa3bf19fab31bbe35"
format_stamp: "Formatted at 2026-09-26 06:29:20 on dist-test-slave-fkbn"
server_key: "676abb365bdac9325c728892a9f0b67a"
server_key_iv: "3c93cd2305753738aa5cb87d8af55d1e"
server_key_version: "encryptionkey@0"
I20260926 06:29:20.852039 24074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "0bf08391de1a496fa3bf19fab31bbe35"
format_stamp: "Formatted at 2026-09-26 06:29:20 on dist-test-slave-fkbn"
server_key: "676abb365bdac9325c728892a9f0b67a"
server_key_iv: "3c93cd2305753738aa5cb87d8af55d1e"
server_key_version: "encryptionkey@0"
I20260926 06:29:20.853328 24074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:29:20.854117 24089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:20.854287 24074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:20.854384 24074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "0bf08391de1a496fa3bf19fab31bbe35"
format_stamp: "Formatted at 2026-09-26 06:29:20 on dist-test-slave-fkbn"
server_key: "676abb365bdac9325c728892a9f0b67a"
server_key_iv: "3c93cd2305753738aa5cb87d8af55d1e"
server_key_version: "encryptionkey@0"
I20260926 06:29:20.854490 24074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:20.864586 24074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:20.866469 24074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:20.866626 24074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:20.870369 24074 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:34355
I20260926 06:29:20.870401 24141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:34355 every 8 connection(s)
I20260926 06:29:20.870741 24074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:20.871198 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 24074
I20260926 06:29:20.871321 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:20.871491 2207 external_mini_cluster.cc:1428] Setting key 4d40911c71f0e3187658a2b883da9c50
I20260926 06:29:20.871685 24142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:20.874964 24142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35: Bootstrap starting.
I20260926 06:29:20.875662 24142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:20.875983 24142 log.cc:824] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:20.876669 24142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35: No bootstrap required, opened a new log
Sep 26 06:29:20 dist-test-slave-fkbn krb5kdc[24061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:29:20.878711 24142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf08391de1a496fa3bf19fab31bbe35" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34355 } }
I20260926 06:29:20.878883 24142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:20.878924 24142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf08391de1a496fa3bf19fab31bbe35, State: Initialized, Role: FOLLOWER
I20260926 06:29:20.879104 24142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf08391de1a496fa3bf19fab31bbe35" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34355 } }
I20260926 06:29:20.879202 24142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:20.879247 24142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:20.879294 24142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:20.879794 24142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf08391de1a496fa3bf19fab31bbe35" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34355 } }
I20260926 06:29:20.879911 24142 leader_election.cc:302] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bf08391de1a496fa3bf19fab31bbe35; no voters:
I20260926 06:29:20.880108 24142 leader_election.cc:288] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:20.880162 24147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:20.880314 24147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [term 1 LEADER]: Becoming Leader. State: Replica: 0bf08391de1a496fa3bf19fab31bbe35, State: Running, Role: LEADER
I20260926 06:29:20.880359 24142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:20.880474 24147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf08391de1a496fa3bf19fab31bbe35" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34355 } }
I20260926 06:29:20.880867 24149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bf08391de1a496fa3bf19fab31bbe35. Latest consensus state: current_term: 1 leader_uuid: "0bf08391de1a496fa3bf19fab31bbe35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf08391de1a496fa3bf19fab31bbe35" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34355 } } }
I20260926 06:29:20.880995 24149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:20.880863 24148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bf08391de1a496fa3bf19fab31bbe35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bf08391de1a496fa3bf19fab31bbe35" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 34355 } } }
I20260926 06:29:20.881307 24148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:20.881477 24155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:20.882046 24155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:20.882246 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:20.872015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59935 (local address 127.2.39.254:34355)
0926 06:29:20.872202 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:20.872210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:20.872234 (+ 24us) server_negotiation.cc:407] Connection header received
0926 06:29:20.872754 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:20.872760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:20.872803 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:20.872912 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:20.873696 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:20.874570 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:20.875623 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:20.876044 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:20.876154 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:20.878472 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:20.878477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:20.878478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:20.878489 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:20.880380 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:20.880781 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:20.880783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:20.880784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:20.880890 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:20.881138 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:20.881139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:20.881139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:20.881446 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:20.881615 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:20.881870 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 06:29:20.882020 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260926 06:29:20.883422 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:20.883559 24155 catalog_manager.cc:1347] Generated new cluster ID: b5274564bb864d85a4712f5c731a8ce4
I20260926 06:29:20.883592 24155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:20.890982 24155 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:20.891503 24155 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:20.896605 24145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:20.885646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35435 (local address 127.2.39.254:34355)
0926 06:29:20.889635 (+ 3989us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:20.889637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:20.889644 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:29:20.889663 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:20.889664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:20.889672 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:20.889730 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:20.890398 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:20.891056 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:20.892145 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:20.892444 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:20.892517 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:20.893917 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:20.893918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:20.893919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:20.893925 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:20.895490 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:20.895861 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:20.895862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:20.895862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:20.895914 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:20.896107 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:20.896108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:20.896108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:20.896218 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:20.896307 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:20.896432 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 06:29:20.896520 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3972}
I20260926 06:29:20.903209 24155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bf08391de1a496fa3bf19fab31bbe35: Generated new TSK 0
I20260926 06:29:20.903409 24155 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:20.911126 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 24074
2026-09-26T06:29:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:20 dist-test-slave-fkbn krb5kdc[24173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:20 dist-test-slave-fkbn krb5kdc[24173](info): set up 2 sockets
Sep 26 06:29:20 dist-test-slave-fkbn krb5kdc[24173](info): commencing operation
krb5kdc: starting...
W20260926 06:29:21.945719 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:22 dist-test-slave-fkbn krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:29:22.064668 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36655
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:36655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:22.146561 24186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:22.146816 24186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:22.146840 24186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:22.148038 24186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:22.148082 24186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:22.148095 24186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:22.148144 24186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:22.148175 24186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:22.149729 24186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:36655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:36655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:22.149987 24186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:22.150218 24186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:22.155265 24191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:22.155372 24192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:22.155319 24186 server_base.cc:1034] running on GCE node
W20260926 06:29:22.155299 24194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:22.155792 24186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:22.156121 24186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:22.157298 24186 hybrid_clock.cc:648] HybridClock initialized: now 1790404162157272 us; error 38 us; skew 500 ppm
Sep 26 06:29:22 dist-test-slave-fkbn krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404162, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:22.159878 24186 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:29:22.160327 24186 webserver.cc:460] Webserver started at http://127.2.39.254:39895/ using document root <none> and password file <none>
I20260926 06:29:22.160508 24186 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:22.160547 24186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:22.160635 24186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:22.161767 24186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "2bfc77095c1d4509b107fd2f88784f3a"
format_stamp: "Formatted at 2026-09-26 06:29:22 on dist-test-slave-fkbn"
server_key: "cde51b75a123ceed942b845fa221c04b"
server_key_iv: "94d3305850183fb3a5b262b401ef1a22"
server_key_version: "encryptionkey@0"
I20260926 06:29:22.162084 24186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "2bfc77095c1d4509b107fd2f88784f3a"
format_stamp: "Formatted at 2026-09-26 06:29:22 on dist-test-slave-fkbn"
server_key: "cde51b75a123ceed942b845fa221c04b"
server_key_iv: "94d3305850183fb3a5b262b401ef1a22"
server_key_version: "encryptionkey@0"
I20260926 06:29:22.163380 24186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:29:22.164045 24201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:22.164232 24186 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 06:29:22.164312 24186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "2bfc77095c1d4509b107fd2f88784f3a"
format_stamp: "Formatted at 2026-09-26 06:29:22 on dist-test-slave-fkbn"
server_key: "cde51b75a123ceed942b845fa221c04b"
server_key_iv: "94d3305850183fb3a5b262b401ef1a22"
server_key_version: "encryptionkey@0"
I20260926 06:29:22.164366 24186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:22.178340 24186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:22.180677 24186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:22.180794 24186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:22.184641 24186 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:36655
I20260926 06:29:22.184718 24253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:36655 every 8 connection(s)
I20260926 06:29:22.185067 24186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:22.186118 24254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:22.188509 24254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a: Bootstrap starting.
I20260926 06:29:22.189116 24254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:22.189440 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 24186
I20260926 06:29:22.189491 24254 log.cc:824] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:22.189558 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:22.189782 2207 external_mini_cluster.cc:1428] Setting key e7cf315f8b09e4c7be01ae75880bea61
I20260926 06:29:22.190460 24254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a: No bootstrap required, opened a new log
I20260926 06:29:22.193361 24254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfc77095c1d4509b107fd2f88784f3a" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36655 } }
I20260926 06:29:22.193547 24254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:22.193604 24254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bfc77095c1d4509b107fd2f88784f3a, State: Initialized, Role: FOLLOWER
I20260926 06:29:22.193727 24254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfc77095c1d4509b107fd2f88784f3a" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36655 } }
I20260926 06:29:22.193831 24254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:22.193876 24254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:22.193925 24254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:22.194489 24254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfc77095c1d4509b107fd2f88784f3a" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36655 } }
I20260926 06:29:22.194625 24254 leader_election.cc:302] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bfc77095c1d4509b107fd2f88784f3a; no voters:
I20260926 06:29:22.194772 24254 leader_election.cc:288] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:22.194859 24259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:22.195055 24259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [term 1 LEADER]: Becoming Leader. State: Replica: 2bfc77095c1d4509b107fd2f88784f3a, State: Running, Role: LEADER
I20260926 06:29:22.195039 24254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:22.195191 24259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfc77095c1d4509b107fd2f88784f3a" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36655 } }
I20260926 06:29:22.195655 24261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bfc77095c1d4509b107fd2f88784f3a. Latest consensus state: current_term: 1 leader_uuid: "2bfc77095c1d4509b107fd2f88784f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfc77095c1d4509b107fd2f88784f3a" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36655 } } }
I20260926 06:29:22.195667 24260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bfc77095c1d4509b107fd2f88784f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bfc77095c1d4509b107fd2f88784f3a" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 36655 } } }
I20260926 06:29:22.195789 24260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:22.195996 24261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:22.195997 24267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:22.196689 24267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:22.197497 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:22.190475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34949 (local address 127.2.39.254:36655)
0926 06:29:22.190656 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:22.190666 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:22.190703 (+ 37us) server_negotiation.cc:407] Connection header received
0926 06:29:22.191720 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:22.191730 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:22.191797 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:22.191970 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:22.192191 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:22.193687 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:22.194575 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:22.195224 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:22.195390 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:22.196640 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:22.196644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:22.196645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:22.196849 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:22.196891 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:22.196906 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:22.196981 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:22.197211 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 06:29:22.197281 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260926 06:29:22.198444 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:22.198487 24267 catalog_manager.cc:1347] Generated new cluster ID: 19e8260903844b7db316f3f6f328b091
I20260926 06:29:22.198521 24267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:22.203524 24258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:22.199881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54713 (local address 127.2.39.254:36655)
0926 06:29:22.199922 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:22.199924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:22.200075 (+ 151us) server_negotiation.cc:407] Connection header received
0926 06:29:22.200135 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:22.200136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:22.200146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:22.200215 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:22.200343 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:22.201091 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:22.202133 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:22.202445 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:22.202520 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:22.203268 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:22.203269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:22.203270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:22.203275 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:22.203295 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:22.203301 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:22.203368 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:22.203425 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 06:29:22.203447 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:29:22.205808 24267 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:22.206470 24267 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:22.209633 24267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bfc77095c1d4509b107fd2f88784f3a: Generated new TSK 0
I20260926 06:29:22.209803 24267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:22.216354 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 24186
2026-09-26T06:29:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:22 dist-test-slave-fkbn krb5kdc[24285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:22 dist-test-slave-fkbn krb5kdc[24285](info): set up 2 sockets
Sep 26 06:29:22 dist-test-slave-fkbn krb5kdc[24285](info): commencing operation
krb5kdc: starting...
W20260926 06:29:23.245754 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:23 dist-test-slave-fkbn krb5kdc[24285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:29:23.369248 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:46411
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:46411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:23.450462 24298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:23.450747 24298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:23.450790 24298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:23.452045 24298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:23.452137 24298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:23.452160 24298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:23.452183 24298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:23.452209 24298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:23.453773 24298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:46411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:46411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:23.454077 24298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:23.454320 24298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:23.459496 24303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:23.459650 24298 server_base.cc:1034] running on GCE node
W20260926 06:29:23.459470 24306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:23.459466 24304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:23.460071 24298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:23.460330 24298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:23.461484 24298 hybrid_clock.cc:648] HybridClock initialized: now 1790404163461460 us; error 41 us; skew 500 ppm
Sep 26 06:29:23 dist-test-slave-fkbn krb5kdc[24285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404163, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:23.463961 24298 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:29:23.464421 24298 webserver.cc:460] Webserver started at http://127.2.39.254:39625/ using document root <none> and password file <none>
I20260926 06:29:23.464676 24298 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:23.464723 24298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:23.464851 24298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:23.466008 24298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "50038eb9348d4050b1a9d29599b7f85e"
format_stamp: "Formatted at 2026-09-26 06:29:23 on dist-test-slave-fkbn"
server_key: "ac446e05eb2476556fcc28a77002c4ad"
server_key_iv: "9b9a36c75f4e34202bf7f57b59b1cbe2"
server_key_version: "encryptionkey@0"
I20260926 06:29:23.466361 24298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "50038eb9348d4050b1a9d29599b7f85e"
format_stamp: "Formatted at 2026-09-26 06:29:23 on dist-test-slave-fkbn"
server_key: "ac446e05eb2476556fcc28a77002c4ad"
server_key_iv: "9b9a36c75f4e34202bf7f57b59b1cbe2"
server_key_version: "encryptionkey@0"
I20260926 06:29:23.467690 24298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:29:23.468410 24313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:23.468781 24298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:29:23.468870 24298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "50038eb9348d4050b1a9d29599b7f85e"
format_stamp: "Formatted at 2026-09-26 06:29:23 on dist-test-slave-fkbn"
server_key: "ac446e05eb2476556fcc28a77002c4ad"
server_key_iv: "9b9a36c75f4e34202bf7f57b59b1cbe2"
server_key_version: "encryptionkey@0"
I20260926 06:29:23.469062 24298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:23.472539 24298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:23.474567 24298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:23.474706 24298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:23.478823 24298 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:46411
I20260926 06:29:23.478856 24365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:46411 every 8 connection(s)
I20260926 06:29:23.479216 24298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:23.480299 24366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:23.482781 24366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e: Bootstrap starting.
I20260926 06:29:23.483386 24366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:23.483673 24366 log.cc:824] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:23.483999 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 24298
I20260926 06:29:23.484135 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:23.484329 2207 external_mini_cluster.cc:1428] Setting key 866e442fc10e5c7f45e6028d5a28ee87
I20260926 06:29:23.484578 24366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e: No bootstrap required, opened a new log
I20260926 06:29:23.486874 24366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50038eb9348d4050b1a9d29599b7f85e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46411 } }
I20260926 06:29:23.487058 24366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 06:29:23.487097 24366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50038eb9348d4050b1a9d29599b7f85e, State: Initialized, Role: FOLLOWER
I20260926 06:29:23.487390 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:23.484928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60925 (local address 127.2.39.254:46411)
0926 06:29:23.485120 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:23.485131 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:23.485165 (+ 34us) server_negotiation.cc:407] Connection header received
0926 06:29:23.485980 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:23.485989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:23.486027 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:23.486180 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:23.486895 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:23.486899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:23.486899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:23.486904 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:23.486949 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:23.486963 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:23.487038 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:23.487132 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 06:29:23.487175 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260926 06:29:23.487257 24366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50038eb9348d4050b1a9d29599b7f85e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46411 } }
I20260926 06:29:23.487661 24366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 06:29:23.487736 24366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 06:29:23.487798 24366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:23.488444 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:23.488544 24366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50038eb9348d4050b1a9d29599b7f85e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46411 } }
I20260926 06:29:23.488643 24366 leader_election.cc:302] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50038eb9348d4050b1a9d29599b7f85e; no voters:
I20260926 06:29:23.488806 24366 leader_election.cc:288] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 06:29:23.488874 24371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:23.489019 24371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [term 1 LEADER]: Becoming Leader. State: Replica: 50038eb9348d4050b1a9d29599b7f85e, State: Running, Role: LEADER
I20260926 06:29:23.489063 24366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:23.489176 24371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50038eb9348d4050b1a9d29599b7f85e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46411 } }
I20260926 06:29:23.489600 24371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50038eb9348d4050b1a9d29599b7f85e. Latest consensus state: current_term: 1 leader_uuid: "50038eb9348d4050b1a9d29599b7f85e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50038eb9348d4050b1a9d29599b7f85e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46411 } } }
I20260926 06:29:23.489823 24371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:23.489741 24373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50038eb9348d4050b1a9d29599b7f85e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50038eb9348d4050b1a9d29599b7f85e" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46411 } } }
I20260926 06:29:23.490077 24373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:23.490391 24384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:23.490942 24384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:23.491636 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:23.490158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44971 (local address 127.2.39.254:46411)
0926 06:29:23.490192 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:23.490194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:23.490486 (+ 292us) server_negotiation.cc:407] Connection header received
0926 06:29:23.490576 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:23.490577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:23.490582 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:23.490656 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:23.491281 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:23.491283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:23.491283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 06:29:23.491288 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:23.491302 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 06:29:23.491307 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:23.491349 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:23.491551 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 06:29:23.491565 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:29:23.492817 24384 catalog_manager.cc:1347] Generated new cluster ID: 62bdf9063c074bed9030916067e4dcec
I20260926 06:29:23.492867 24384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:23.497493 24384 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:23.497676 24384 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:23.503333 24384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50038eb9348d4050b1a9d29599b7f85e: Generated new TSK 0
I20260926 06:29:23.503566 24384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 06:29:23.513258 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 24298
2026-09-26T06:29:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1297 ms)
[----------] 18 tests from AuthTokenIssuingTest (23721 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:23 dist-test-slave-fkbn krb5kdc[24397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:23 dist-test-slave-fkbn krb5kdc[24397](info): set up 2 sockets
Sep 26 06:29:23 dist-test-slave-fkbn krb5kdc[24397](info): commencing operation
krb5kdc: starting...
W20260926 06:29:24.549799 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:24 dist-test-slave-fkbn krb5kdc[24397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:29:24.672636 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:46865
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:46865,127.2.39.253:35639,127.2.39.252:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:24.753809 24410 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:29:24.754025 24410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:24.754043 24410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:24.754065 24410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:24.755153 24410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:24.755189 24410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:24.755203 24410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:24.755215 24410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:24.755228 24410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:24.756655 24410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:46865,127.2.39.253:35639,127.2.39.252:40867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:46865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:24.756934 24410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:24.757144 24410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:24.762483 24416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:24.762480 24415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:24.762571 24410 server_base.cc:1034] running on GCE node
W20260926 06:29:24.762511 24418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:24.762974 24410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:24.763316 24410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:24.764492 24410 hybrid_clock.cc:648] HybridClock initialized: now 1790404164764481 us; error 32 us; skew 500 ppm
Sep 26 06:29:24 dist-test-slave-fkbn krb5kdc[24397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:24.773437 24410 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:29:24.773993 24410 webserver.cc:460] Webserver started at http://127.2.39.254:41453/ using document root <none> and password file <none>
I20260926 06:29:24.774217 24410 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:24.774261 24410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:24.774354 24410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:24.775404 24410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "0bff31bf96284265ab1b8dd81163cb4d"
format_stamp: "Formatted at 2026-09-26 06:29:24 on dist-test-slave-fkbn"
server_key: "95d6f2f434da79621167bc00ea67beeb"
server_key_iv: "d275fc78e231cc46f726b1778cfba1a5"
server_key_version: "encryptionkey@0"
I20260926 06:29:24.775699 24410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "0bff31bf96284265ab1b8dd81163cb4d"
format_stamp: "Formatted at 2026-09-26 06:29:24 on dist-test-slave-fkbn"
server_key: "95d6f2f434da79621167bc00ea67beeb"
server_key_iv: "d275fc78e231cc46f726b1778cfba1a5"
server_key_version: "encryptionkey@0"
I20260926 06:29:24.777022 24410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:29:24.777951 24425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:24.778173 24410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:29:24.778266 24410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "0bff31bf96284265ab1b8dd81163cb4d"
format_stamp: "Formatted at 2026-09-26 06:29:24 on dist-test-slave-fkbn"
server_key: "95d6f2f434da79621167bc00ea67beeb"
server_key_iv: "d275fc78e231cc46f726b1778cfba1a5"
server_key_version: "encryptionkey@0"
I20260926 06:29:24.778334 24410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:24.797523 24410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:24.800004 24410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:24.800118 24410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:24.804003 24410 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:46865
I20260926 06:29:24.804086 24477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:46865 every 8 connection(s)
I20260926 06:29:24.804440 24410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:24.805473 24478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:24.807051 24478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:24.807773 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 24410
I20260926 06:29:24.807896 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:24.808089 2207 external_mini_cluster.cc:1428] Setting key bffcd8de1ef053483b4d962ac04d94c1
Sep 26 06:29:24 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.39.253@KRBTEST.COM; defaulting to no policy
I20260926 06:29:24.818219 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.807721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46865 (local address 127.0.0.1:41384)
0926 06:29:24.807988 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.808038 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:24.808765 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:24.809093 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:24.809096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:24.809493 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:24.809764 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:24.809770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.810815 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.810816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:24.811542 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:24.811545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.812391 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.812393 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:24.812470 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:24.813000 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:24.813012 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:24.814682 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:24.816775 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:24.816781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:24.816783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:24.816980 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:24.817197 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:24.817198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:24.817198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:24.817277 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:24.817732 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:24.817738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:24.817798 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 06:29:24.817907 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 06:29:24.818133 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":81,"thread_start_us":147,"threads_started":1}
I20260926 06:29:24.818223 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.807807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.2.39.254:46865)
0926 06:29:24.807995 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:24.808005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:24.808158 (+ 153us) server_negotiation.cc:407] Connection header received
0926 06:29:24.808952 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:24.808954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:24.808971 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:24.809062 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:24.809889 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.810696 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.811653 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.812273 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.812392 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:24.814773 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:24.814780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:24.814781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:24.814789 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:24.816694 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:24.817058 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:24.817060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:24.817061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:24.817136 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:24.817335 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:24.817335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:24.817336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:24.817660 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:24.817727 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:24.817955 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 06:29:24.818052 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
I20260926 06:29:24.819065 24478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.253@KRBTEST.COM" created.
I20260926 06:29:24.819468 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.819321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:35639 (local address 127.0.0.1:58230)
0926 06:29:24.819374 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.819441 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 06:29:24.819612 24428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.39.253:35639: connect: Connection refused (error 111)
I20260926 06:29:24.820035 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.819911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:35639 (local address 127.0.0.1:58242)
0926 06:29:24.819979 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.820007 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 06:29:24.820147 24478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.253:35639: Network error: Client connection negotiation failed: client connection to 127.2.39.253:35639: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab.
Entry for principal kudu/127.2.39.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab.
I20260926 06:29:24.836699 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.253:35639
--webserver_interface=127.2.39.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab
--principal=kudu/127.2.39.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:46865,127.2.39.253:35639,127.2.39.252:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:29:24.869415 24478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } attempt: 1
I20260926 06:29:24.869983 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.869845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:35639 (local address 127.0.0.1:58246)
0926 06:29:24.869911 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.869953 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 06:29:24.870402 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.870290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:35639 (local address 127.0.0.1:58250)
0926 06:29:24.870344 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.870376 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 06:29:24.870533 24478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.253:35639: Network error: Client connection negotiation failed: client connection to 127.2.39.253:35639: connect: Connection refused (error 111)
W20260926 06:29:24.919298 24485 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:29:24.919576 24485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:24.919608 24485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:24.919648 24485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:24.920783 24485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:24.920861 24485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:24.920884 24485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:24.920907 24485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:24.920929 24485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:24.922456 24485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:46865,127.2.39.253:35639,127.2.39.252:40867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab
--principal=kudu/127.2.39.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.253:35639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.39.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:24.922747 24485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:24.922977 24485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:24.928100 24491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:24.928138 24490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:24.928138 24493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:24.928164 24485 server_base.cc:1034] running on GCE node
I20260926 06:29:24.928635 24485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:24.928895 24485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:24.930096 24485 hybrid_clock.cc:648] HybridClock initialized: now 1790404164930080 us; error 45 us; skew 500 ppm
Sep 26 06:29:24 dist-test-slave-fkbn krb5kdc[24397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:24.932724 24485 init.cc:377] Logged in from keytab as kudu/127.2.39.253@KRBTEST.COM (short username kudu)
I20260926 06:29:24.933228 24485 webserver.cc:460] Webserver started at http://127.2.39.253:43563/ using document root <none> and password file <none>
I20260926 06:29:24.933475 24485 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:24.933547 24485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:24.933736 24485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:24.934803 24485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/data/instance:
uuid: "db6f08421bec4019878c1f8d8b54f57a"
format_stamp: "Formatted at 2026-09-26 06:29:24 on dist-test-slave-fkbn"
server_key: "bdfc1b95fce4632f939fc45f4dbc4c6a"
server_key_iv: "18c32f90b5fa13082c4da8a67bca2ecf"
server_key_version: "encryptionkey@0"
I20260926 06:29:24.935192 24485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/wal/instance:
uuid: "db6f08421bec4019878c1f8d8b54f57a"
format_stamp: "Formatted at 2026-09-26 06:29:24 on dist-test-slave-fkbn"
server_key: "bdfc1b95fce4632f939fc45f4dbc4c6a"
server_key_iv: "18c32f90b5fa13082c4da8a67bca2ecf"
server_key_version: "encryptionkey@0"
I20260926 06:29:24.936556 24485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 06:29:24.937340 24500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:24.937527 24485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:29:24.937652 24485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/wal
uuid: "db6f08421bec4019878c1f8d8b54f57a"
format_stamp: "Formatted at 2026-09-26 06:29:24 on dist-test-slave-fkbn"
server_key: "bdfc1b95fce4632f939fc45f4dbc4c6a"
server_key_iv: "18c32f90b5fa13082c4da8a67bca2ecf"
server_key_version: "encryptionkey@0"
I20260926 06:29:24.937734 24485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:24.938706 24478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } attempt: 2
I20260926 06:29:24.939203 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.939048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:35639 (local address 127.0.0.1:58266)
0926 06:29:24.939126 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.939169 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 06:29:24.939651 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.939542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:35639 (local address 127.0.0.1:58274)
0926 06:29:24.939594 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.939620 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:35639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 06:29:24.939735 24478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.253:35639: Network error: Client connection negotiation failed: client connection to 127.2.39.253:35639: connect: Connection refused (error 111)
I20260926 06:29:24.955621 24485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:24.957772 24485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:24.957928 24485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:24.962669 24485 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.253:35639
I20260926 06:29:24.962713 24552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.253:35639 every 8 connection(s)
I20260926 06:29:24.963095 24485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
I20260926 06:29:24.964151 24553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:24.965869 24553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:29:24 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:29:24.971984 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 24485
I20260926 06:29:24.972112 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-1/wal/instance
I20260926 06:29:24.972535 2207 external_mini_cluster.cc:1428] Setting key 97d631bfd6ce4905b9b5ee7567966640
I20260926 06:29:24.975761 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.966689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.2.39.254:46865)
0926 06:29:24.966762 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:24.966764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:24.966961 (+ 197us) server_negotiation.cc:407] Connection header received
0926 06:29:24.967429 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:24.967431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:24.967440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:24.967503 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:24.968085 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.968587 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.969699 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.969958 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.970025 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:24.972323 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:24.972324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:24.972325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:24.972334 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:24.974222 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:24.974682 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:24.974684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:24.974684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:24.974750 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:24.975072 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:24.975073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:24.975073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:24.975196 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:24.975276 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:24.975577 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 06:29:24.975665 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:29:24.975963 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.966673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46865 (local address 127.0.0.1:41390)
0926 06:29:24.966828 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.966872 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:24.967338 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:24.967558 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:24.967561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:24.967805 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:24.968009 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:24.968013 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.968675 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.968676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:24.969516 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:24.969519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.970054 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.970055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:24.970169 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:24.970717 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:24.970726 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:24.972170 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:24.974336 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:24.974345 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:24.974348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:24.974584 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:24.974843 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:24.974844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:24.974844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:24.974968 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:24.975269 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:24.975277 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:24.975432 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 06:29:24.975544 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 06:29:24.975722 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260926 06:29:24.976774 24553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.39.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.252@KRBTEST.COM" created.
Sep 26 06:29:24 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
I20260926 06:29:24.987879 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.977124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58286 (local address 127.2.39.253:35639)
0926 06:29:24.977287 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:24.977292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:24.977319 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:29:24.977346 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:24.977348 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:24.977360 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:24.977422 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:24.977821 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.978546 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.979592 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.979911 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.979975 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:24.984620 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:24.984633 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:24.984633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:24.984642 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:24.986409 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:24.986887 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:24.986890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:24.986890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:24.986955 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:24.987197 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:24.987198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:24.987198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:24.987377 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:24.987454 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:24.987684 (+ 230us) server_negotiation.cc:299] Negotiation successful
0926 06:29:24.987768 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20260926 06:29:24.987983 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.977071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:35639 (local address 127.0.0.1:58286)
0926 06:29:24.977133 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.977146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:24.977197 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:24.977451 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:24.977452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:24.977646 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:24.977714 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:24.977716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.978644 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.978645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:24.979493 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:24.979496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:24.980003 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:24.980004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:24.980093 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:24.980489 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:24.980494 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:24.984528 (+ 4034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:24.986504 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:24.986505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:24.986506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:24.986733 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:24.987039 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:24.987040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:24.987040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:24.987111 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:24.987462 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:24.987463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:24.987515 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 06:29:24.987598 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:29:24.987708 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 06:29:24.988693 24553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:24.989094 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.988947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54618)
0926 06:29:24.989002 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.989062 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 06:29:24.989284 24501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
I20260926 06:29:24.989784 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:24.989680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54620)
0926 06:29:24.989728 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:24.989760 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 06:29:24.989908 24553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:40867: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab.
Entry for principal kudu/127.2.39.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab.
I20260926 06:29:25.001462 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.252:40867
--webserver_interface=127.2.39.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab
--principal=kudu/127.2.39.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:45275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:46865,127.2.39.253:35639,127.2.39.252:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:29:25.030913 24478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } attempt: 3
Sep 26 06:29:25 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
I20260926 06:29:25.038654 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.031358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:35639 (local address 127.0.0.1:58294)
0926 06:29:25.031425 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.031443 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:25.031513 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:25.031676 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:25.031676 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:25.031852 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:25.031921 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.031923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.032649 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.032651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.033216 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.033218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.033681 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.033682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.033764 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.034297 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:25.034301 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:25.035635 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:25.037468 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.037469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.037470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.037728 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.037962 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.037963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.037963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.038024 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.038301 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:25.038302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:25.038352 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 06:29:25.038441 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 06:29:25.038547 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 06:29:25.038654 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.031429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.2.39.253:35639)
0926 06:29:25.031505 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.031507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.031518 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:29:25.031572 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.031573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.031597 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.031669 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.032012 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.032540 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.033316 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.033559 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.033674 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.035736 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.035737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.035738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.035746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.037349 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.037820 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.037822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.037822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.037889 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.038098 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.038099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.038100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.038221 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.038303 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.038465 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.038549 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 06:29:25.039140 24553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } attempt: 1
I20260926 06:29:25.039367 24478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:25.039590 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.039413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54624)
0926 06:29:25.039500 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.039546 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 06:29:25.039716 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.039591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54628)
0926 06:29:25.039656 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.039686 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 06:29:25.039973 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.039881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54642)
0926 06:29:25.039918 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.039941 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 06:29:25.040050 24553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:40867: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
I20260926 06:29:25.040159 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.040019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54654)
0926 06:29:25.040114 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.040138 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260926 06:29:25.040263 24478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:40867: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
I20260926 06:29:25.071467 24478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } attempt: 1
I20260926 06:29:25.072005 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.071844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54666)
0926 06:29:25.071916 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.071961 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 06:29:25.072373 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.072283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54674)
0926 06:29:25.072318 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.072349 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 06:29:25.072477 24478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:40867: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
W20260926 06:29:25.086532 24560 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:29:25.086753 24560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:25.086771 24560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:25.086792 24560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:25.088007 24560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:25.088055 24560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:25.088070 24560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:25.088083 24560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:25.088096 24560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:25.089700 24560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:45275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:46865,127.2.39.253:35639,127.2.39.252:40867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab
--principal=kudu/127.2.39.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.252:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.39.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:25.089968 24560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:25.090226 24560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:25.095263 24565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:25.095258 24566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:25.095276 24560 server_base.cc:1034] running on GCE node
W20260926 06:29:25.095258 24568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:25.095724 24560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:25.095984 24560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:25.097155 24560 hybrid_clock.cc:648] HybridClock initialized: now 1790404165097142 us; error 28 us; skew 500 ppm
Sep 26 06:29:25 dist-test-slave-fkbn krb5kdc[24397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404165, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:25.099896 24560 init.cc:377] Logged in from keytab as kudu/127.2.39.252@KRBTEST.COM (short username kudu)
I20260926 06:29:25.100312 24560 webserver.cc:460] Webserver started at http://127.2.39.252:45577/ using document root <none> and password file <none>
I20260926 06:29:25.100498 24560 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:25.100539 24560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:25.100629 24560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:25.101831 24560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/data/instance:
uuid: "e4de4e83219c453d8290956fab70ab3c"
format_stamp: "Formatted at 2026-09-26 06:29:25 on dist-test-slave-fkbn"
server_key: "9d507321d9ec7377d9d4c828aea39600"
server_key_iv: "4d3163c148c5143a0c4ef449d4158211"
server_key_version: "encryptionkey@0"
I20260926 06:29:25.102149 24560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/wal/instance:
uuid: "e4de4e83219c453d8290956fab70ab3c"
format_stamp: "Formatted at 2026-09-26 06:29:25 on dist-test-slave-fkbn"
server_key: "9d507321d9ec7377d9d4c828aea39600"
server_key_iv: "4d3163c148c5143a0c4ef449d4158211"
server_key_version: "encryptionkey@0"
I20260926 06:29:25.103489 24560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:29:25.104189 24575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:25.104389 24560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:29:25.104501 24560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/wal
uuid: "e4de4e83219c453d8290956fab70ab3c"
format_stamp: "Formatted at 2026-09-26 06:29:25 on dist-test-slave-fkbn"
server_key: "9d507321d9ec7377d9d4c828aea39600"
server_key_iv: "4d3163c148c5143a0c4ef449d4158211"
server_key_version: "encryptionkey@0"
I20260926 06:29:25.104555 24560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:25.108192 24553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } attempt: 2
I20260926 06:29:25.108660 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.108524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54684)
0926 06:29:25.108590 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.108630 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 06:29:25.109030 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.108939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54696)
0926 06:29:25.108985 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.109012 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 06:29:25.109140 24553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:40867: Network error: Client connection negotiation failed: client connection to 127.2.39.252:40867: connect: Connection refused (error 111)
I20260926 06:29:25.125540 24560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:25.128036 24560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:25.128209 24560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:25.132184 24560 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.252:40867
I20260926 06:29:25.132222 24627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.252:40867 every 8 connection(s)
I20260926 06:29:25.132632 24560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
I20260926 06:29:25.133818 24628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:25.135456 24628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:25.136654 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 24560
I20260926 06:29:25.136736 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790403899360596-2207-0/minicluster-data/master-2/wal/instance
I20260926 06:29:25.136958 2207 external_mini_cluster.cc:1428] Setting key b77a590bf3c6595df3fee2028489bc2a
Sep 26 06:29:25 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404165, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
Sep 26 06:29:25 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
I20260926 06:29:25.145932 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.136196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.2.39.254:46865)
0926 06:29:25.136242 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.136244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.136471 (+ 227us) server_negotiation.cc:407] Connection header received
0926 06:29:25.137084 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.137085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.137098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.137171 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.138025 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.138704 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.139930 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.140247 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.140316 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.142965 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.142967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.142967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.142975 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.144492 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.144918 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.144919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.144919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.144980 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.145233 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.145234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.145234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.145352 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.145439 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.145757 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.145836 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:29:25.146128 24630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.136148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:46865 (local address 127.0.0.1:41396)
0926 06:29:25.136312 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.136361 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:25.136947 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:25.137258 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:25.137263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:25.137668 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:25.137936 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.137940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.138782 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.138783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.139846 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.139848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.140359 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.140360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.140462 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.140977 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:25.140987 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:25.142881 (+ 1894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:25.144583 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.144589 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.144591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.144822 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.145059 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.145059 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.145060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.145165 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.145422 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:25.145429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:25.145607 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 06:29:25.145736 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 06:29:25.145882 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":73,"threads_started":1}
I20260926 06:29:25.146658 24628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:25.147008 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.137336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54704 (local address 127.2.39.252:40867)
0926 06:29:25.137522 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.137526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.137601 (+ 75us) server_negotiation.cc:407] Connection header received
0926 06:29:25.137687 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.137690 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.137715 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.137792 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.138479 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.139632 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.140641 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.141076 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.141150 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.144066 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.144074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.144074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.144081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.145677 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.146074 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.146078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.146078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.146146 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.146401 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.146402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.146403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.146591 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.146717 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.146826 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.146907 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":130,"threads_started":1}
I20260926 06:29:25.147593 24478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } attempt: 2
Sep 26 06:29:25 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404165, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
Sep 26 06:29:25 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
Sep 26 06:29:25 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:29:25.154321 24630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.146915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:35639 (local address 127.0.0.1:58298)
0926 06:29:25.147031 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.147042 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:25.147090 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:25.147210 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:25.147211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:25.147356 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:25.147413 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.147414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.148283 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.148284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.149043 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.149045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.149532 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.149532 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.149625 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.150019 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:25.150024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:25.151256 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:25.153119 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.153121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.153121 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.153355 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.153624 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.153625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.153626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.153689 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.153997 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:25.153998 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:25.154042 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 06:29:25.154116 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:29:25.154205 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 06:29:25.154371 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.146985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.2.39.253:35639)
0926 06:29:25.147029 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.147031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.147076 (+ 45us) server_negotiation.cc:407] Connection header received
0926 06:29:25.147151 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.147152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.147161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.147209 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.147526 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.148218 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.149134 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.149416 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.149458 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.151355 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.151356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.151357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.151364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.153040 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.153474 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.153475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.153475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.153534 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.153766 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.153767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.153768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.153918 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.153989 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.154172 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.154246 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:29:25.154992 24628 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:25.155117 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.147821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54712)
0926 06:29:25.147886 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.147897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:25.147945 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:25.148125 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:25.148126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:25.148358 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:25.148408 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.148410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.149232 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.149233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.149902 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.149905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.150319 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.150320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.150374 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.150775 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:25.150779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:25.152008 (+ 1229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:25.153814 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.153815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.153815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.154037 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.154512 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.154514 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.154514 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.154581 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.154819 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:25.154819 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:25.154865 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 06:29:25.154947 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 06:29:25.155057 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 06:29:25.155421 24632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.147894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.2.39.252:40867)
0926 06:29:25.147927 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.147928 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.147937 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:29:25.148054 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.148055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.148064 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.148104 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.148498 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.149156 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.149993 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.150238 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.150515 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.152098 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.152099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.152099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.152106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.153751 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.154118 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.154119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.154120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.154404 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.154652 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.154653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.154653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.154762 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.154824 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.155240 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.155316 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:29:25.157344 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.148013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.2.39.254:46865)
0926 06:29:25.148049 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.148051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.148059 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:29:25.148150 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.148151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.148159 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.148202 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.148779 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.149246 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.150210 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.150771 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.150836 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.153919 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.153920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.153920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.153927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.155930 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.156306 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.156308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.156308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.156367 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.156579 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.156579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.156580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.156689 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.157169 (+ 480us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.157203 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.157263 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 06:29:25.157755 24478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d: Bootstrap starting.
I20260926 06:29:25.158465 24478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:25.158795 24478 log.cc:824] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:25.159677 24478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d: No bootstrap required, opened a new log
Sep 26 06:29:25 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404165, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
I20260926 06:29:25.162012 24478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } }
I20260926 06:29:25.162161 24478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bff31bf96284265ab1b8dd81163cb4d, State: Initialized, Role: FOLLOWER
I20260926 06:29:25.162272 24478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } }
I20260926 06:29:25.162567 24635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } } }
I20260926 06:29:25.162662 24478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:25.162726 24635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:25.163818 24630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.155274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54724)
0926 06:29:25.155571 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.155581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:25.155630 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:25.155792 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:25.155792 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:25.155936 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:25.155970 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.155972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.156789 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.156790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.157539 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.157542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.158036 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.158037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.158082 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.158569 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:25.158574 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:25.160116 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:25.162608 (+ 2492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.162609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.162609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.162837 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.163059 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.163060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.163060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.163112 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.163484 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:25.163485 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:25.163528 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 06:29:25.163602 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 06:29:25.163693 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260926 06:29:25.164264 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.155336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.2.39.252:40867)
0926 06:29:25.155537 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.155539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.155642 (+ 103us) server_negotiation.cc:407] Connection header received
0926 06:29:25.155669 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.155670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.155682 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.155754 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.156042 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.156700 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.157674 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.157959 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.158026 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.160203 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.160204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.160205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.160212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.162515 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.162919 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.162921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.162921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.162991 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.163169 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.163170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.163171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.163370 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.163459 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.164046 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.164131 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":112,"threads_started":1}
Sep 26 06:29:25 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
W20260926 06:29:25.164819 24646 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:29:25.164862 24646 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:25.166234 24628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c: Bootstrap starting.
I20260926 06:29:25.166982 24628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:25.167268 24628 log.cc:824] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:25.168123 24628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c: No bootstrap required, opened a new log
I20260926 06:29:25.165668 24410 master_runner.cc:186] Error getting master registration for 127.2.39.253:35639: OK, instance_id { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" instance_seqno: 1790404164961270 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:29:25.169008 24410 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:29:25.169291 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.159667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.2.39.253:35639)
0926 06:29:25.159735 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.159737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.159746 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:29:25.159770 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.159771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.159782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.159841 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.160495 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.161176 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.162199 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.162535 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.162597 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.166065 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.166066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.166067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.166074 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.167862 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.168264 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.168265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.168265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.168322 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.168525 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.168526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.168526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.168648 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.168779 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.169160 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.169229 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:29:25.170203 24628 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } }
I20260926 06:29:25.170358 24628 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4de4e83219c453d8290956fab70ab3c, State: Initialized, Role: FOLLOWER
I20260926 06:29:25.170463 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:25.170442 24628 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } }
I20260926 06:29:25.170746 24628 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:25.170722 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } } }
I20260926 06:29:25.170807 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:25.171614 24452 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0bff31bf96284265ab1b8dd81163cb4d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41406
I20260926 06:29:25.171737 24452 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 06:29:25.171785 24452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [term 0 FOLLOWER]: Advancing to term 1
W20260926 06:29:25.172515 24660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 06:29:25.172727 24452 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } }
I20260926 06:29:25.172969 24452 leader_election.cc:288] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [CANDIDATE]: Term 1 election: Requested vote from peers db6f08421bec4019878c1f8d8b54f57a (127.2.39.253:35639), e4de4e83219c453d8290956fab70ab3c (127.2.39.252:40867)
W20260926 06:29:25.172694 24660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:25.173888 24602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bff31bf96284265ab1b8dd81163cb4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e4de4e83219c453d8290956fab70ab3c"
I20260926 06:29:25.174011 24602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:25.174906 24602 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bff31bf96284265ab1b8dd81163cb4d in term 1.
I20260926 06:29:25.175200 24426 leader_election.cc:302] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bff31bf96284265ab1b8dd81163cb4d, e4de4e83219c453d8290956fab70ab3c; no voters:
I20260926 06:29:25.175331 24635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:25.175467 24635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [term 1 LEADER]: Becoming Leader. State: Replica: 0bff31bf96284265ab1b8dd81163cb4d, State: Running, Role: LEADER
I20260926 06:29:25.175559 24635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } }
I20260926 06:29:25.175984 24635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bff31bf96284265ab1b8dd81163cb4d. Latest consensus state: current_term: 1 leader_uuid: "0bff31bf96284265ab1b8dd81163cb4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } } }
I20260926 06:29:25.176072 24635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:25.176857 24668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:25.177261 24668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:25.177224 24560 master_runner.cc:186] Error getting master registration for 127.2.39.253:35639: OK, instance_id { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" instance_seqno: 1790404164961270 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:29:25.179584 24560 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:29:25.186857 24602 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [term 1 FOLLOWER]: Refusing update from remote peer 0bff31bf96284265ab1b8dd81163cb4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:29:25.187216 24666 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:29:25.188753 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bff31bf96284265ab1b8dd81163cb4d. Latest consensus state: current_term: 1 leader_uuid: "0bff31bf96284265ab1b8dd81163cb4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } } }
I20260926 06:29:25.188839 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:25.195569 24666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bff31bf96284265ab1b8dd81163cb4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } } }
I20260926 06:29:25.195691 24666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:25.196348 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bff31bf96284265ab1b8dd81163cb4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } } }
I20260926 06:29:25.196453 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:25.196445 24668 catalog_manager.cc:1347] Generated new cluster ID: c7d009b36ea34acfaeddd5ef8aae96ef
I20260926 06:29:25.196488 24668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:25.196662 24673 mvcc.cc:205] Tried to move back new op lower bound from 7333495460601536512 to 7333495460559859712. Current Snapshot: MvccSnapshot[applied={T|T < 7333495460601536512}]
I20260926 06:29:25.197463 24480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.186109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.2.39.254:46865)
0926 06:29:25.186153 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.186166 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.187201 (+ 1035us) server_negotiation.cc:407] Connection header received
0926 06:29:25.187381 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.187382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.187392 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.187421 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.188198 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.188903 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.189970 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.190267 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.190308 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.191945 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.191947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.191947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.191954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.193857 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.194271 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.194273 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.194273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.194332 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.194823 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.194824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.194825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.194984 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.195078 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.195557 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.195642 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:29:25.199648 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.185955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.2.39.252:40867)
0926 06:29:25.186004 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.186006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.186913 (+ 907us) server_negotiation.cc:407] Connection header received
0926 06:29:25.187310 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.187311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.187323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.187376 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.188716 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.189487 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.191307 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.191658 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.191700 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.194155 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.194157 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.194157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.194164 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.197282 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.198490 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.198506 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.198506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.198564 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.198879 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.198880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.198893 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.199022 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.199125 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.199461 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.199557 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 06:29:25.200347 24553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } attempt: 3
Sep 26 06:29:25 dist-test-slave-fkbn krb5kdc[24397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
I20260926 06:29:25.210171 24555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.200578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:40867 (local address 127.0.0.1:54744)
0926 06:29:25.200669 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:25.200680 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:25.200719 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:25.201296 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:25.201297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:25.201465 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:25.201519 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.201521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.202445 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.202446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.203225 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:25.203228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.203722 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.203723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:25.203776 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.204291 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:25.204296 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:25.205823 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:25.208597 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.208599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.208599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.208853 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.209213 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:25.209214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:25.209214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:25.209285 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:25.209818 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:25.209819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:25.209882 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 06:29:25.209944 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 06:29:25.210061 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 06:29:25.210855 24633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.201104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.2.39.252:40867)
0926 06:29:25.201169 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.201170 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.201178 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:29:25.201211 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.201212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.201222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.201273 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.201622 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.202354 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.203310 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.203647 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.203723 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.205939 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.205940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.205941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.205951 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.208494 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.208942 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.208943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.208943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.209079 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.209370 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.209371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.209371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.209681 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.210458 (+ 777us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.210533 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.210678 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 06:29:25.212627 24553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a: Bootstrap starting.
I20260926 06:29:25.213397 24553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:25.213879 24553 log.cc:824] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:25.221585 24553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a: No bootstrap required, opened a new log
I20260926 06:29:25.223626 24553 raft_consensus.cc:348] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } }
I20260926 06:29:25.223783 24553 raft_consensus.cc:729] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6f08421bec4019878c1f8d8b54f57a, State: Initialized, Role: FOLLOWER
I20260926 06:29:25.223876 24553 consensus_queue.cc:260] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } }
I20260926 06:29:25.224260 24553 sys_catalog.cc:564] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:25.224354 24526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:25.225289 24526 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [term 1 FOLLOWER]: Refusing update from remote peer 0bff31bf96284265ab1b8dd81163cb4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:29:25.225509 24527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0bff31bf96284265ab1b8dd81163cb4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "db6f08421bec4019878c1f8d8b54f57a"
I20260926 06:29:25.224237 24679 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } } }
I20260926 06:29:25.225806 24666 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:29:25.225960 24679 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:25.227017 24527 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0bff31bf96284265ab1b8dd81163cb4d in term 1.
I20260926 06:29:25.227916 24668 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:25.228684 24668 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:25.228796 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:25.186609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.2.39.253:35639)
0926 06:29:25.186651 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.186654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.194726 (+ 8072us) server_negotiation.cc:407] Connection header received
0926 06:29:25.194851 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.194852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.194866 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.194899 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.196839 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.197558 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.199912 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:25.200238 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:25.200296 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:25.220198 (+ 19902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:25.220200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:25.220200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:25.220209 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:25.222032 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.227803 (+ 5771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.227805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.227805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.227885 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:25.228178 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:25.228179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:25.228180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:25.228311 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:25.228390 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:25.228617 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 06:29:25.228706 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:29:25.234139 24679 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bff31bf96284265ab1b8dd81163cb4d. Latest consensus state: current_term: 1 leader_uuid: "0bff31bf96284265ab1b8dd81163cb4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } } }
I20260926 06:29:25.234227 24679 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:25.234264 24679 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0bff31bf96284265ab1b8dd81163cb4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } } }
W20260926 06:29:25.234292 24690 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:29:25.234338 24690 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:25.234301 24679 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:25.238749 24691 mvcc.cc:205] Tried to move back new op lower bound from 7333495460735287296 to 7333495460559859712. Current Snapshot: MvccSnapshot[applied={T|T < 7333495460601536512}]
I20260926 06:29:25.247982 24666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0bff31bf96284265ab1b8dd81163cb4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0bff31bf96284265ab1b8dd81163cb4d" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 46865 } } peers { permanent_uuid: "db6f08421bec4019878c1f8d8b54f57a" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 35639 } } peers { permanent_uuid: "e4de4e83219c453d8290956fab70ab3c" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 40867 } } }
I20260926 06:29:25.248129 24666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:25.249327 24668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bff31bf96284265ab1b8dd81163cb4d: Generated new TSK 0
I20260926 06:29:25.249549 24668 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:29:25.269251 24480 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.267932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.2.39.254:46865)
0926 06:29:25.267979 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.267981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.268558 (+ 577us) server_negotiation.cc:407] Connection header received
0926 06:29:25.268665 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.268666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.268677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.268708 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.269176 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41428: BlockingRecv error: recv got EOF from 127.0.0.1:41428 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 06:29:25.270262 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.268266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58334 (local address 127.2.39.253:35639)
0926 06:29:25.268306 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.268308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.269725 (+ 1417us) server_negotiation.cc:407] Connection header received
0926 06:29:25.269864 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.269865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.269876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.269907 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.270202 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58334: BlockingRecv error: recv got EOF from 127.0.0.1:58334 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 06:29:25.271044 24633 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.269536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.2.39.252:40867)
0926 06:29:25.269592 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.269594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.270464 (+ 870us) server_negotiation.cc:407] Connection header received
0926 06:29:25.270623 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.270624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.270634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.270662 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.270987 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54746: BlockingRecv error: recv got EOF from 127.0.0.1:54746 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 06:29:25.273267 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.272524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.2.39.253:35639)
0926 06:29:25.272577 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.272579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.272774 (+ 195us) server_negotiation.cc:407] Connection header received
0926 06:29:25.272879 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.272880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.272899 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.272930 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.273219 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58348: BlockingRecv error: recv got EOF from 127.0.0.1:58348 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 06:29:25.276175 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.275417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58360 (local address 127.2.39.253:35639)
0926 06:29:25.275448 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.275450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.275706 (+ 256us) server_negotiation.cc:407] Connection header received
0926 06:29:25.275798 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.275799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.275808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.275836 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.276129 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58360: BlockingRecv error: recv got EOF from 127.0.0.1:58360 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 06:29:25.280741 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.279872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58374 (local address 127.2.39.253:35639)
0926 06:29:25.279907 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.279908 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.280136 (+ 228us) server_negotiation.cc:407] Connection header received
0926 06:29:25.280228 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.280229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.280239 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.280266 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.280692 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58374: BlockingRecv error: recv got EOF from 127.0.0.1:58374 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 06:29:25.287362 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.286570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58384 (local address 127.2.39.253:35639)
0926 06:29:25.286607 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.286608 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.286848 (+ 240us) server_negotiation.cc:407] Connection header received
0926 06:29:25.286937 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.286938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.286949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.286978 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.287297 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58384: BlockingRecv error: recv got EOF from 127.0.0.1:58384 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 06:29:25.297418 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.296539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58390 (local address 127.2.39.253:35639)
0926 06:29:25.296580 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.296583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.296841 (+ 258us) server_negotiation.cc:407] Connection header received
0926 06:29:25.296944 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.296945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.296956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.296986 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.297348 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58390: BlockingRecv error: recv got EOF from 127.0.0.1:58390 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 06:29:25.315627 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.314710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58392 (local address 127.2.39.253:35639)
0926 06:29:25.314754 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.314756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.315040 (+ 284us) server_negotiation.cc:407] Connection header received
0926 06:29:25.315145 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.315146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.315157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.315186 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.315565 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58392: BlockingRecv error: recv got EOF from 127.0.0.1:58392 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 06:29:25.349982 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.348969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.2.39.253:35639)
0926 06:29:25.349018 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.349020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.349325 (+ 305us) server_negotiation.cc:407] Connection header received
0926 06:29:25.349456 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.349457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.349469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.349501 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.349919 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58406: BlockingRecv error: recv got EOF from 127.0.0.1:58406 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 06:29:25.416144 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.415509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58408 (local address 127.2.39.253:35639)
0926 06:29:25.415582 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.415583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.415594 (+ 11us) server_negotiation.cc:407] Connection header received
0926 06:29:25.415668 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.415669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.415680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.415722 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.416084 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58408: BlockingRecv error: recv got EOF from 127.0.0.1:58408 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 06:29:25.546259 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.545410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.2.39.253:35639)
0926 06:29:25.545496 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.545498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.545700 (+ 202us) server_negotiation.cc:407] Connection header received
0926 06:29:25.545751 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.545752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.545763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.545838 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.546176 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58410: BlockingRecv error: recv got EOF from 127.0.0.1:58410 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 06:29:25.804247 24557 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:25.803575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58414 (local address 127.2.39.253:35639)
0926 06:29:25.803646 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:25.803648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:25.803658 (+ 10us) server_negotiation.cc:407] Connection header received
0926 06:29:25.803683 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:25.803684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:25.803694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:25.803746 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:25.804194 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58414: BlockingRecv error: recv got EOF from 127.0.0.1:58414 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 06:29:26.173646 24660 catalog_manager.cc:1259] Loaded cluster ID: c7d009b36ea34acfaeddd5ef8aae96ef
I20260926 06:29:26.173733 24660 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c: loading cluster ID for follower catalog manager: success
I20260926 06:29:26.174569 24660 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c: acquiring CA information for follower catalog manager: success
I20260926 06:29:26.174889 24660 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4de4e83219c453d8290956fab70ab3c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:29:26.234782 24690 catalog_manager.cc:1259] Loaded cluster ID: c7d009b36ea34acfaeddd5ef8aae96ef
I20260926 06:29:26.234872 24690 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a: loading cluster ID for follower catalog manager: success
I20260926 06:29:26.235718 24690 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a: acquiring CA information for follower catalog manager: success
I20260926 06:29:26.236068 24690 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db6f08421bec4019878c1f8d8b54f57a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:29:26.320050 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:26.317467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.2.39.253:35639)
0926 06:29:26.317753 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:26.317755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:26.317764 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:29:26.317790 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:26.317790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:26.317842 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:26.317932 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:29:26.318090 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:26.318584 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:26.319299 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:26.319507 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:26.319569 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:26.319706 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:29:26.319814 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:29:26.319882 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:26.319904 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 06:29:26.319922 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":71,"threads_started":1}
I20260926 06:29:26.324606 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:26.321948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.2.39.252:40867)
0926 06:29:26.322044 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:26.322046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:26.322073 (+ 27us) server_negotiation.cc:407] Connection header received
0926 06:29:26.322147 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:26.322148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:26.322182 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:26.322246 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:29:26.322407 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:26.322896 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:26.323683 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:26.323866 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:26.323916 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:26.324091 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:29:26.324192 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:29:26.324276 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:26.324474 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 06:29:26.324499 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":53,"threads_started":1}
I20260926 06:29:26.329108 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:26.326558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.2.39.252:40867)
0926 06:29:26.326596 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:26.326598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:26.326607 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:29:26.326661 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:26.326662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:26.326673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:26.326862 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:29:26.326893 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:26.327450 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:26.328305 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:26.328613 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:26.328682 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:26.328829 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:29:26.328863 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:29:26.328922 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:26.328994 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 06:29:26.329015 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 06:29:26.329807 24774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:26.326492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.2.39.254:46865)
0926 06:29:26.326629 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:26.326631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:26.326730 (+ 99us) server_negotiation.cc:407] Connection header received
0926 06:29:26.326847 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:26.326848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:26.326901 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:26.326971 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:29:26.327159 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:26.327910 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:26.328760 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:26.329074 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:26.329134 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:26.329436 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:29:26.329572 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:29:26.329655 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:26.329700 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 06:29:26.329722 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260926 06:29:26.329974 24761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:26.326755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58434 (local address 127.2.39.253:35639)
0926 06:29:26.326794 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:26.326796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:26.326805 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:29:26.326831 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:26.326832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:26.326845 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:26.326901 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:29:26.327037 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:26.327786 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:26.328806 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:26.329098 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:26.329135 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:26.329600 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:29:26.329637 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:29:26.329691 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:26.329866 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 06:29:26.329892 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:29:26.331502 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 24410
I20260926 06:29:26.337433 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 24485
I20260926 06:29:26.342365 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 24560
2026-09-26T06:29:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2830 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 06:29:26 dist-test-slave-fkbn krb5kdc[24778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 06:29:26 dist-test-slave-fkbn krb5kdc[24778](info): set up 2 sockets
Sep 26 06:29:26 dist-test-slave-fkbn krb5kdc[24778](info): commencing operation
krb5kdc: starting...
W20260926 06:29:27.386003 2207 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/test-user.keytab.
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T06:29:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T06:29:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.39.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
Entry for principal kudu/127.2.39.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab.
I20260926 06:29:27.509064 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:42369
--webserver_interface=127.2.39.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:42369,127.2.39.253:42003,127.2.39.252:38769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 06:29:27.591998 24791 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:29:27.592338 24791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:27.592365 24791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:27.592413 24791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:27.594403 24791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:27.594455 24791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:27.594488 24791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:27.594511 24791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:27.594532 24791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:27.596598 24791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:42369,127.2.39.253:42003,127.2.39.252:38769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.254.keytab
--principal=kudu/127.2.39.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.254:42369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.39.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:27.596906 24791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:27.597173 24791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:27.602677 24797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:27.602732 24799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:27.602864 24796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:27.602742 24791 server_base.cc:1034] running on GCE node
I20260926 06:29:27.603139 24791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:27.603475 24791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:27.604645 24791 hybrid_clock.cc:648] HybridClock initialized: now 1790404167604628 us; error 26 us; skew 500 ppm
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:27.607707 24791 init.cc:377] Logged in from keytab as kudu/127.2.39.254@KRBTEST.COM (short username kudu)
I20260926 06:29:27.608192 24791 webserver.cc:460] Webserver started at http://127.2.39.254:42789/ using document root <none> and password file <none>
I20260926 06:29:27.608474 24791 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:27.608522 24791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:27.608692 24791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:27.609855 24791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/data/instance:
uuid: "1b3fc0500e3643678744f4b25408b6a4"
format_stamp: "Formatted at 2026-09-26 06:29:27 on dist-test-slave-fkbn"
server_key: "3b04a471b0b7099b47320bc2bef8b763"
server_key_iv: "efa5d2efcf9e35957c1af9804e409671"
server_key_version: "encryptionkey@0"
I20260926 06:29:27.610217 24791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/wal/instance:
uuid: "1b3fc0500e3643678744f4b25408b6a4"
format_stamp: "Formatted at 2026-09-26 06:29:27 on dist-test-slave-fkbn"
server_key: "3b04a471b0b7099b47320bc2bef8b763"
server_key_iv: "efa5d2efcf9e35957c1af9804e409671"
server_key_version: "encryptionkey@0"
I20260926 06:29:27.611573 24791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 06:29:27.612485 24806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:27.612852 24791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 06:29:27.612955 24791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/wal
uuid: "1b3fc0500e3643678744f4b25408b6a4"
format_stamp: "Formatted at 2026-09-26 06:29:27 on dist-test-slave-fkbn"
server_key: "3b04a471b0b7099b47320bc2bef8b763"
server_key_iv: "efa5d2efcf9e35957c1af9804e409671"
server_key_version: "encryptionkey@0"
I20260926 06:29:27.613029 24791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:27.646749 24791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:27.648810 24791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:27.648976 24791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:27.653717 24791 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.254:42369
I20260926 06:29:27.653753 24858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.254:42369 every 8 connection(s)
I20260926 06:29:27.654134 24791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/data/info.pb
I20260926 06:29:27.655205 24859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:27.656816 24859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:29:27.664274 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 24791
I20260926 06:29:27.664403 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-0/wal/instance
I20260926 06:29:27.664633 2207 external_mini_cluster.cc:1428] Setting key 112e8e5b9a9d23b16d1821e894d29d49
I20260926 06:29:27.666803 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.657483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42369 (local address 127.0.0.1:43020)
0926 06:29:27.657669 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.657712 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:27.658159 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:27.658470 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:27.658472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:27.658714 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:27.658867 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.658870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.659929 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.659930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.660646 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.660648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.661171 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.661173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.661231 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.661802 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:27.661810 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:27.663274 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:27.665105 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:27.665112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:27.665114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:27.665386 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:27.665741 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:27.665742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:27.665743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:27.665854 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:27.666370 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:27.666376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:27.666437 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 06:29:27.666514 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 06:29:27.666637 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260926 06:29:27.666880 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.657556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.2.39.254:42369)
0926 06:29:27.657742 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:27.657752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:27.657784 (+ 32us) server_negotiation.cc:407] Connection header received
0926 06:29:27.658290 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:27.658292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:27.658351 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:27.658441 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:27.658955 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.659838 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.660734 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.661041 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.661139 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.663358 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:27.663365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:27.663366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:27.663372 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:27.665026 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:27.665470 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:27.665473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:27.665473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:27.665552 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:27.665968 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:27.665969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:27.665970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:27.666287 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:27.666392 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:27.666577 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 06:29:27.666723 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":154,"thread_start_us":129,"threads_started":1}
I20260926 06:29:27.667586 24859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:27.667937 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.667821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:42003 (local address 127.0.0.1:35206)
0926 06:29:27.667870 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.667911 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 06:29:27.668064 24807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.39.253:42003: connect: Connection refused (error 111)
I20260926 06:29:27.668463 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.668365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:42003 (local address 127.0.0.1:35208)
0926 06:29:27.668420 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.668443 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 06:29:27.668563 24859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.253:42003: Network error: Client connection negotiation failed: client connection to 127.2.39.253:42003: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.39.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab.
Entry for principal kudu/127.2.39.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab.
I20260926 06:29:27.692287 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.253:42003
--webserver_interface=127.2.39.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab
--principal=kudu/127.2.39.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:42369,127.2.39.253:42003,127.2.39.252:38769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:29:27.717767 24859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } attempt: 1
I20260926 06:29:27.718322 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.718166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:42003 (local address 127.0.0.1:35224)
0926 06:29:27.718238 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.718285 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 06:29:27.718770 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.718651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:42003 (local address 127.0.0.1:35232)
0926 06:29:27.718711 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.718743 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 06:29:27.718909 24859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.253:42003: Network error: Client connection negotiation failed: client connection to 127.2.39.253:42003: connect: Connection refused (error 111)
W20260926 06:29:27.774238 24866 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:29:27.774466 24866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:27.774484 24866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:27.774504 24866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:27.775602 24866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:27.775636 24866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:27.775650 24866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:27.775663 24866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:27.775676 24866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:27.777082 24866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:42369,127.2.39.253:42003,127.2.39.252:38769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.253.keytab
--principal=kudu/127.2.39.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.253:42003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.39.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:27.777278 24866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:27.777459 24866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 06:29:27.782407 24874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:27.782423 24871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:27.782423 24872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:27.782653 24866 server_base.cc:1034] running on GCE node
I20260926 06:29:27.782948 24866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:27.783252 24866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:27.784426 24866 hybrid_clock.cc:648] HybridClock initialized: now 1790404167784406 us; error 29 us; skew 500 ppm
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:27.787070 24859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } attempt: 2
I20260926 06:29:27.787153 24866 init.cc:377] Logged in from keytab as kudu/127.2.39.253@KRBTEST.COM (short username kudu)
I20260926 06:29:27.787490 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.787331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:42003 (local address 127.0.0.1:35240)
0926 06:29:27.787404 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.787447 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 06:29:27.787734 24866 webserver.cc:460] Webserver started at http://127.2.39.253:45137/ using document root <none> and password file <none>
I20260926 06:29:27.787900 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.787763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:42003 (local address 127.0.0.1:35244)
0926 06:29:27.787842 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.787876 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.253:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 06:29:27.788014 24859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.253:42003: Network error: Client connection negotiation failed: client connection to 127.2.39.253:42003: connect: Connection refused (error 111)
I20260926 06:29:27.788056 24866 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:27.788128 24866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:27.788271 24866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:27.789350 24866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/data/instance:
uuid: "7e107df297c34572813ef9c895018b52"
format_stamp: "Formatted at 2026-09-26 06:29:27 on dist-test-slave-fkbn"
server_key: "b5f30701d608988f7b80f14fb99ce5be"
server_key_iv: "903267c36494b328cb9b04d6e3885dff"
server_key_version: "encryptionkey@0"
I20260926 06:29:27.789762 24866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/wal/instance:
uuid: "7e107df297c34572813ef9c895018b52"
format_stamp: "Formatted at 2026-09-26 06:29:27 on dist-test-slave-fkbn"
server_key: "b5f30701d608988f7b80f14fb99ce5be"
server_key_iv: "903267c36494b328cb9b04d6e3885dff"
server_key_version: "encryptionkey@0"
I20260926 06:29:27.791146 24866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:29:27.791865 24881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:27.792026 24866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:27.792128 24866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/wal
uuid: "7e107df297c34572813ef9c895018b52"
format_stamp: "Formatted at 2026-09-26 06:29:27 on dist-test-slave-fkbn"
server_key: "b5f30701d608988f7b80f14fb99ce5be"
server_key_iv: "903267c36494b328cb9b04d6e3885dff"
server_key_version: "encryptionkey@0"
I20260926 06:29:27.792200 24866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:27.802807 24866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:27.804598 24866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:27.804775 24866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:27.808755 24866 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.253:42003
I20260926 06:29:27.808784 24933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.253:42003 every 8 connection(s)
I20260926 06:29:27.809156 24866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/data/info.pb
I20260926 06:29:27.810274 24934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:27.811766 24934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:27.817230 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 24866
I20260926 06:29:27.817353 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-1/wal/instance
I20260926 06:29:27.817536 2207 external_mini_cluster.cc:1428] Setting key 9fd92d2bfc22b2a551aadb6593b6cf94
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:29:27.821285 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.812462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.2.39.254:42369)
0926 06:29:27.812532 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:27.812534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:27.812754 (+ 220us) server_negotiation.cc:407] Connection header received
0926 06:29:27.813222 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:27.813223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:27.813236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:27.813300 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:27.813987 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.814529 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.815532 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.815730 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.815798 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.818333 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:27.818335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:27.818335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:27.818344 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:27.819781 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:27.820288 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:27.820289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:27.820289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:27.820342 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:27.820632 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:27.820633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:27.820633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:27.820739 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:27.820789 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:27.821151 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 06:29:27.821219 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 06:29:27.821600 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.812450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42369 (local address 127.0.0.1:43034)
0926 06:29:27.812637 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.812671 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:27.813133 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:27.813370 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:27.813373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:27.813692 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:27.813887 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.813891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.814621 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.814622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.815464 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.815466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.815815 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.815816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.815946 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.816533 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:27.816542 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:27.818214 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:27.819866 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:27.819873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:27.819875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:27.820209 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:27.820418 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:27.820419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:27.820419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:27.820569 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:27.820815 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:27.820822 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:27.821001 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 06:29:27.821131 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 06:29:27.821319 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260926 06:29:27.822228 24934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.39.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.39.252@KRBTEST.COM" created.
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
I20260926 06:29:27.831836 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.822525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:42003 (local address 127.0.0.1:35246)
0926 06:29:27.822586 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.822595 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:27.822638 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:27.822909 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:27.822909 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:27.823058 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:27.823101 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.823103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.823962 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.823963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.824461 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.824463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.824952 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.824953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.825016 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.825464 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:27.825468 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:27.828570 (+ 3102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:27.830771 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:27.830772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:27.830773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:27.830935 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:27.831154 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:27.831154 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:27.831155 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:27.831204 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:27.831540 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:27.831541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:27.831575 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 06:29:27.831630 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 06:29:27.831763 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 06:29:27.832001 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.822602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.2.39.253:42003)
0926 06:29:27.822774 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:27.822782 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:27.822797 (+ 15us) server_negotiation.cc:407] Connection header received
0926 06:29:27.822814 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:27.822816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:27.822826 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:27.822908 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:27.823182 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.823838 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.824552 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.824845 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.824945 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.828656 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:27.828664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:27.828664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:27.828670 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:27.830684 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:27.831017 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:27.831021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:27.831021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:27.831088 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:27.831266 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:27.831267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:27.831268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:27.831470 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:27.831560 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:27.831679 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 06:29:27.831846 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260926 06:29:27.832842 24934 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:27.833209 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.833097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59296)
0926 06:29:27.833148 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.833189 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 06:29:27.833376 24884 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
I20260926 06:29:27.833890 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.833797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59310)
0926 06:29:27.833851 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.833875 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 06:29:27.834024 24934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:38769: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.39.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab.
Entry for principal kudu/127.2.39.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab.
I20260926 06:29:27.844919 2207 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOgNaKF/build/release/bin/kudu
/tmp/dist-test-taskOgNaKF/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.252:38769
--webserver_interface=127.2.39.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab
--principal=kudu/127.2.39.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.39.212:43025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.39.254:42369,127.2.39.253:42003,127.2.39.252:38769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 06:29:27.879171 24859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } attempt: 3
I20260926 06:29:27.883280 24934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } attempt: 1
I20260926 06:29:27.883673 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.883558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59316)
0926 06:29:27.883612 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.883647 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
I20260926 06:29:27.884017 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.883899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59322)
0926 06:29:27.883965 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.883992 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 06:29:27.884159 24934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:38769: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
I20260926 06:29:27.886830 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.879572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:42003 (local address 127.0.0.1:35260)
0926 06:29:27.879629 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.879646 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:27.879688 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:27.879883 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:27.879884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:27.880055 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:27.880119 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.880121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.880812 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.880813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.881361 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.881363 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.881849 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.881850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.881936 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.882425 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:27.882430 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:27.884085 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:27.885908 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:27.885909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:27.885909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:27.886075 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:27.886289 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:27.886289 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:27.886289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:27.886340 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:27.886612 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:27.886613 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:27.886651 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 06:29:27.886699 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 06:29:27.886770 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 06:29:27.886915 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.879683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.2.39.253:42003)
0926 06:29:27.879754 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:27.879756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:27.879765 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:29:27.879792 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:27.879793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:27.879804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:27.879876 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:27.880207 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.880719 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.881453 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.881755 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.881847 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.884205 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:27.884206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:27.884207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:27.884216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:27.885823 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:27.886149 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:27.886150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:27.886151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:27.886218 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:27.886401 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:27.886402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:27.886402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:27.886544 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:27.886618 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:27.886737 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 06:29:27.886827 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:29:27.887286 24859 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:27.887634 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.887524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59324)
0926 06:29:27.887577 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.887614 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 06:29:27.887988 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.887876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59330)
0926 06:29:27.887929 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.887962 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 06:29:27.888094 24859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:38769: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
I20260926 06:29:27.919267 24859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } attempt: 1
I20260926 06:29:27.919802 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.919622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59342)
0926 06:29:27.919706 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.919762 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 06:29:27.920217 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.920104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59352)
0926 06:29:27.920157 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.920191 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 06:29:27.920326 24859 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:38769: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
W20260926 06:29:27.932576 24941 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 06:29:27.932814 24941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 06:29:27.932835 24941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 06:29:27.932914 24941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 06:29:27.934165 24941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 06:29:27.934214 24941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 06:29:27.934231 24941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 06:29:27.934291 24941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 06:29:27.934314 24941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 06:29:27.935981 24941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.39.212:43025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.39.254:42369,127.2.39.253:42003,127.2.39.252:38769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/krb5kdc/kudu_127.2.39.252.keytab
--principal=kudu/127.2.39.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.39.252:38769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.39.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 05:33:27 UTC on ccycloud.quasar-vdapsj.root.comops.site
build id 4673735
I20260926 06:29:27.936272 24941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 06:29:27.936542 24941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 06:29:27.941790 24941 server_base.cc:1034] running on GCE node
W20260926 06:29:27.941726 24946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:27.941753 24947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 06:29:27.941735 24949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 06:29:27.942183 24941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 06:29:27.942467 24941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 06:29:27.943619 24941 hybrid_clock.cc:648] HybridClock initialized: now 1790404167943604 us; error 32 us; skew 500 ppm
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 06:29:27.946280 24941 init.cc:377] Logged in from keytab as kudu/127.2.39.252@KRBTEST.COM (short username kudu)
I20260926 06:29:27.946823 24941 webserver.cc:460] Webserver started at http://127.2.39.252:39355/ using document root <none> and password file <none>
I20260926 06:29:27.947077 24941 fs_manager.cc:359] Metadata directory not provided
I20260926 06:29:27.947124 24941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 06:29:27.947273 24941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 06:29:27.948385 24941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/data/instance:
uuid: "acc593f2d16b40ea820f2dd4a4a197ab"
format_stamp: "Formatted at 2026-09-26 06:29:27 on dist-test-slave-fkbn"
server_key: "bf7b82fa27c13569a14c8b0600f4545a"
server_key_iv: "9c7d6750e4db6f4bfae9882a9806ecd0"
server_key_version: "encryptionkey@0"
I20260926 06:29:27.948745 24941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/wal/instance:
uuid: "acc593f2d16b40ea820f2dd4a4a197ab"
format_stamp: "Formatted at 2026-09-26 06:29:27 on dist-test-slave-fkbn"
server_key: "bf7b82fa27c13569a14c8b0600f4545a"
server_key_iv: "9c7d6750e4db6f4bfae9882a9806ecd0"
server_key_version: "encryptionkey@0"
I20260926 06:29:27.950191 24941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 06:29:27.951066 24956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 06:29:27.951275 24941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 06:29:27.951370 24941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/data,/tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/wal
uuid: "acc593f2d16b40ea820f2dd4a4a197ab"
format_stamp: "Formatted at 2026-09-26 06:29:27 on dist-test-slave-fkbn"
server_key: "bf7b82fa27c13569a14c8b0600f4545a"
server_key_iv: "9c7d6750e4db6f4bfae9882a9806ecd0"
server_key_version: "encryptionkey@0"
I20260926 06:29:27.951444 24941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 06:29:27.952293 24934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } attempt: 2
I20260926 06:29:27.952685 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.952562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59360)
0926 06:29:27.952621 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.952652 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 06:29:27.953128 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.953028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59368)
0926 06:29:27.953085 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.953110 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 06:29:27.953248 24934 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.39.252:38769: Network error: Client connection negotiation failed: client connection to 127.2.39.252:38769: connect: Connection refused (error 111)
I20260926 06:29:27.975296 24941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 06:29:27.977242 24941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 06:29:27.977438 24941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 06:29:27.981925 24941 rpc_server.cc:307] RPC server started. Bound to: 127.2.39.252:38769
I20260926 06:29:27.981962 25008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.39.252:38769 every 8 connection(s)
I20260926 06:29:27.982350 24941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/data/info.pb
I20260926 06:29:27.983335 25009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 06:29:27.984833 25009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:27.989938 2207 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOgNaKF/build/release/bin/kudu as pid 24941
I20260926 06:29:27.990046 2207 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOgNaKF/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790403899360596-2207-0/minicluster-data/master-2/wal/instance
I20260926 06:29:27.990278 2207 external_mini_cluster.cc:1428] Setting key 9551a8d00deb1f438b66a12c2ade7e70
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
I20260926 06:29:27.994881 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.985517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.2.39.254:42369)
0926 06:29:27.985646 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:27.985648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:27.985846 (+ 198us) server_negotiation.cc:407] Connection header received
0926 06:29:27.986356 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:27.986358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:27.986369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:27.986443 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:27.987065 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.987543 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.988552 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.988744 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.988833 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.991596 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:27.991598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:27.991598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:27.991604 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:27.993338 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:27.993859 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:27.993861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:27.993861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:27.993956 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:27.994242 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:27.994242 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:27.994243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:27.994365 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:27.994413 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:27.994725 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 06:29:27.994816 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 06:29:27.995157 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.985504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.254:42369 (local address 127.0.0.1:43038)
0926 06:29:27.985710 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.985754 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:27.986246 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:27.986514 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:27.986518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:27.986767 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:27.986965 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.986969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.987639 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.987640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.988458 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.988460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.988855 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.988856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.988995 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.989503 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:27.989511 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:27.991507 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:27.993449 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:27.993455 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:27.993457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:27.993779 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:27.994029 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:27.994029 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:27.994030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:27.994161 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:27.994411 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:27.994416 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:27.994580 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 06:29:27.994677 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 06:29:27.994857 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":111,"threads_started":1}
I20260926 06:29:27.995453 24859 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } attempt: 2
I20260926 06:29:27.995831 25009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.254@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
Sep 26 06:29:27 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
I20260926 06:29:28.001626 25012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.990703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.2.39.252:38769)
0926 06:29:27.990841 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:27.990846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:27.990984 (+ 138us) server_negotiation.cc:407] Connection header received
0926 06:29:27.991057 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:27.991059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:27.991083 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:27.991155 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:27.991872 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.992743 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.993962 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.994443 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.994516 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.997421 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:27.997428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:27.997429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:27.997435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:27.999815 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.000238 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.000241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.000241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.000297 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.000536 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.000537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.000538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.000733 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:28.000906 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:28.001131 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 06:29:28.001284 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260926 06:29:28.003616 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.996109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.253:42003 (local address 127.0.0.1:35268)
0926 06:29:27.996138 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.996148 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:27.996181 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:27.996392 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:27.996392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:27.996542 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:27.996584 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.996585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.997469 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.997470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.998175 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.998177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.998682 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.998683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.998768 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.999079 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:27.999083 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:28.000791 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:28.002509 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:28.002510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:28.002511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:28.002723 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:28.002952 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:28.002954 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:28.002954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:28.003020 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:28.003313 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:28.003314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:28.003353 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 06:29:28.003436 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:29:28.003526 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 06:29:28.003633 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.996231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.2.39.253:42003)
0926 06:29:27.996277 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:27.996279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:27.996287 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:29:27.996313 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:27.996313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:27.996325 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:27.996376 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:27.996676 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.997373 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.998277 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.998588 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.998660 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.000880 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:28.000882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:28.000882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:28.000889 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:28.002398 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.002811 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.002812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.002812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.002879 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.003104 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.003105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.003105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.003249 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:28.003324 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:28.003471 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 06:29:28.003546 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:29:28.003751 24861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.995695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59396)
0926 06:29:27.995730 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:27.995741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:27.995787 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:27.996084 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:27.996085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:27.996239 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:27.996285 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.996287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.997088 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.997089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.997792 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:27.997794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.998182 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.998183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:27.998244 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:27.998673 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:27.998677 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:28.000247 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:28.002610 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:28.002611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:28.002611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:28.002784 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:28.003021 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:28.003022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:28.003022 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:28.003084 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:28.003379 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:28.003380 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:28.003436 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 06:29:28.003519 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 06:29:28.003624 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 06:29:28.004091 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:27.995794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59396 (local address 127.2.39.252:38769)
0926 06:29:27.995938 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:27.995940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:27.995953 (+ 13us) server_negotiation.cc:407] Connection header received
0926 06:29:27.995990 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:27.995991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:27.996003 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:27.996078 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:27.996374 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.997010 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.997874 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:27.998095 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:27.998180 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.000350 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:28.000351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:28.000351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:28.000356 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:28.002529 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.002872 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.002873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.002874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.002942 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.003163 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.003164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.003165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.003284 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:28.003373 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:28.003892 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 06:29:28.003971 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260926 06:29:28.004554 25009 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } has no permanent_uuid. Determining permanent_uuid...
I20260926 06:29:28.006865 24859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4: Bootstrap starting.
I20260926 06:29:28.007656 24859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:28.008023 24859 log.cc:824] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4: Log is configured to *not* fsync() on all Append() calls
Sep 26 06:29:28 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.254@KRBTEST.COM
Sep 26 06:29:28 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.252@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
I20260926 06:29:28.012004 24859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4: No bootstrap required, opened a new log
I20260926 06:29:28.013564 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:28.003085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43052 (local address 127.2.39.254:42369)
0926 06:29:28.003122 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.003124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.003130 (+ 6us) server_negotiation.cc:407] Connection header received
0926 06:29:28.003148 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.003149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.003159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.003211 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.003912 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.004647 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.005812 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.006217 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.006355 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.010844 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:28.010845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:28.010845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:28.010851 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:28.012351 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.012777 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.012779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.012779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.012835 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.013053 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.013054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.013054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.013168 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:28.013336 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:28.013375 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 06:29:28.013464 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:29:28.013648 25011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:28.004821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59402)
0926 06:29:28.004897 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:28.004906 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:28.004941 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:28.005913 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:28.005914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:28.006073 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:28.006117 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:28.006119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.006987 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.006988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:28.007702 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:28.007705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.008224 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.008225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:28.008288 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.008747 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:28.008752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:28.010404 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:28.012453 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:28.012454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:28.012454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:28.012658 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:28.012861 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:28.012862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:28.012863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:28.012937 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:28.013314 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:28.013315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:28.013353 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 06:29:28.013404 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 06:29:28.013500 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 06:29:28.014038 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:28.004887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.2.39.252:38769)
0926 06:29:28.005793 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.005795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.005803 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:29:28.005822 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.005823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.005833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.006204 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.006233 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.006887 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.007814 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.008127 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.008435 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.010513 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:28.010515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:28.010515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:28.010522 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:28.012354 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.012742 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.012743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.012743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.012802 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.013016 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.013017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.013017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.013205 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:28.013830 (+ 625us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:28.013874 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 06:29:28.013952 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":886}
I20260926 06:29:28.014523 24859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } }
I20260926 06:29:28.014694 24859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b3fc0500e3643678744f4b25408b6a4, State: Initialized, Role: FOLLOWER
I20260926 06:29:28.014827 24859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } }
I20260926 06:29:28.015257 24859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:28.015609 25009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab: Bootstrap starting.
I20260926 06:29:28.016075 25016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } } }
I20260926 06:29:28.016302 25016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:28.016330 25009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:28.016628 25009 log.cc:824] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab: Log is configured to *not* fsync() on all Append() calls
W20260926 06:29:28.017318 25027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:29:28.017421 25027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 06:29:28 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.39.253@KRBTEST.COM
I20260926 06:29:28.020650 25009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab: No bootstrap required, opened a new log
I20260926 06:29:28.018250 24791 master_runner.cc:186] Error getting master registration for 127.2.39.253:42003: OK, instance_id { permanent_uuid: "7e107df297c34572813ef9c895018b52" instance_seqno: 1790404167807646 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:29:28.021651 24791 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:29:28.022716 25009 raft_consensus.cc:348] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } }
I20260926 06:29:28.022843 25009 raft_consensus.cc:729] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acc593f2d16b40ea820f2dd4a4a197ab, State: Initialized, Role: FOLLOWER
I20260926 06:29:28.022909 25009 consensus_queue.cc:260] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } }
I20260926 06:29:28.023167 25030 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } } }
I20260926 06:29:28.023231 25009 sys_catalog.cc:564] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:28.023241 25030 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:28.024294 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:28.014203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.2.39.253:42003)
0926 06:29:28.014242 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.014243 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.014317 (+ 74us) server_negotiation.cc:407] Connection header received
0926 06:29:28.014411 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.014412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.014425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.014493 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.015113 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.015872 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.016912 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.017248 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.017306 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.020340 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:28.020342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:28.020342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:28.020351 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:28.022797 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.023220 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.023221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.023222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.023278 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.023466 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.023467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.023467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.023596 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:28.023737 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:28.024124 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 06:29:28.024197 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 06:29:28.025418 25041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:29:28.025512 25041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:28.026018 2207 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 06:29:28.026937 24833 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1b3fc0500e3643678744f4b25408b6a4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43052
I20260926 06:29:28.027076 24833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 06:29:28.027120 24833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:28.028335 24833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } }
I20260926 06:29:28.028574 24833 leader_election.cc:288] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [CANDIDATE]: Term 1 election: Requested vote from peers 7e107df297c34572813ef9c895018b52 (127.2.39.253:42003), acc593f2d16b40ea820f2dd4a4a197ab (127.2.39.252:38769)
I20260926 06:29:28.028879 24983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b3fc0500e3643678744f4b25408b6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "acc593f2d16b40ea820f2dd4a4a197ab"
I20260926 06:29:28.028990 24983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:28.026943 24941 master_runner.cc:186] Error getting master registration for 127.2.39.253:42003: OK, instance_id { permanent_uuid: "7e107df297c34572813ef9c895018b52" instance_seqno: 1790404167807646 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 06:29:28.029434 24941 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 06:29:28.030061 24983 raft_consensus.cc:2454] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3fc0500e3643678744f4b25408b6a4 in term 1.
I20260926 06:29:28.030342 24809 leader_election.cc:302] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b3fc0500e3643678744f4b25408b6a4, acc593f2d16b40ea820f2dd4a4a197ab; no voters:
I20260926 06:29:28.030489 25016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 06:29:28.030639 25016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [term 1 LEADER]: Becoming Leader. State: Replica: 1b3fc0500e3643678744f4b25408b6a4, State: Running, Role: LEADER
I20260926 06:29:28.030731 25016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } }
I20260926 06:29:28.031108 25050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b3fc0500e3643678744f4b25408b6a4. Latest consensus state: current_term: 1 leader_uuid: "1b3fc0500e3643678744f4b25408b6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } } }
I20260926 06:29:28.031304 25050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:28.038867 24862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:28.029915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.2.39.254:42369)
0926 06:29:28.029966 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.029968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.030139 (+ 171us) server_negotiation.cc:407] Connection header received
0926 06:29:28.030319 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.030334 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.030345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.030376 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.031251 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.032052 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.033174 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.033473 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.033546 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.035268 (+ 1722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:28.035269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:28.035269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:28.035290 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:28.037121 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.037895 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.037896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.037897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.037957 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.038169 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.038171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.038172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.038329 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:28.038432 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:28.038689 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 06:29:28.038771 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:29:28.044554 24934 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } attempt: 3
I20260926 06:29:28.044931 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:28.030237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.2.39.252:38769)
0926 06:29:28.030276 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.030278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.034057 (+ 3779us) server_negotiation.cc:407] Connection header received
0926 06:29:28.034180 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.034181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.034192 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.034221 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.035842 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.036569 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.037552 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.037891 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.037964 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.041009 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:28.041011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:28.041011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:28.041018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:28.042631 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.043372 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.043373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.043374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.043434 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.044318 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.044319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.044319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.044454 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:28.044549 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:28.044757 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 06:29:28.044839 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:29:28.045744 25052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 06:29:28.045969 25052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 06:29:28.047653 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:28.030428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.2.39.253:42003)
0926 06:29:28.030475 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.030478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.039247 (+ 8769us) server_negotiation.cc:407] Connection header received
0926 06:29:28.039350 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.039352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.039365 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.039425 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.040116 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.040830 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.042449 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.042773 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.042815 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.044181 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:28.044182 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:28.044183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:28.044190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:28.046312 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.046685 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.046686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.046687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.046758 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.046946 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.046947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.046947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.047067 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:28.047150 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:28.047487 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 06:29:28.047562 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 06:29:28 dist-test-slave-fkbn krb5kdc[24778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790404167, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.39.253@KRBTEST.COM for kudu/127.2.39.252@KRBTEST.COM
I20260926 06:29:28.052912 24936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:28.044820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.39.252:38769 (local address 127.0.0.1:59430)
0926 06:29:28.044966 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 06:29:28.044976 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 06:29:28.045019 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 06:29:28.045287 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 06:29:28.045288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 06:29:28.045457 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 06:29:28.045503 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:28.045505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.046561 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.046562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:28.047359 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 06:29:28.047361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.047972 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.047973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 06:29:28.048024 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.048400 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 06:29:28.048405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 06:29:28.049889 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 06:29:28.051736 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:28.051737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:28.051738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:28.051950 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:28.052171 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 06:29:28.052172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 06:29:28.052172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 06:29:28.052229 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 06:29:28.052599 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 06:29:28.052599 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 06:29:28.052646 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 06:29:28.052715 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 06:29:28.052810 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260926 06:29:28.052927 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:28.045095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.2.39.252:38769)
0926 06:29:28.045133 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.045134 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.045142 (+ 8us) server_negotiation.cc:407] Connection header received
0926 06:29:28.045161 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.045162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.045173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.045237 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.045726 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.046480 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.047452 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:28.047861 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:28.047924 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:28.050014 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 06:29:28.050015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 06:29:28.050016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 06:29:28.050023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 06:29:28.051652 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.052039 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.052040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.052040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.052098 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 06:29:28.052385 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 06:29:28.052386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 06:29:28.052386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 06:29:28.052537 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 06:29:28.052602 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:28.052755 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 06:29:28.052831 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 06:29:28.054883 24934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52: Bootstrap starting.
I20260926 06:29:28.055658 24934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52: Neither blocks nor log segments found. Creating new log.
I20260926 06:29:28.055948 24934 log.cc:824] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52: Log is configured to *not* fsync() on all Append() calls
I20260926 06:29:28.057417 24934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52: No bootstrap required, opened a new log
I20260926 06:29:28.059618 24934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } }
I20260926 06:29:28.059765 24934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e107df297c34572813ef9c895018b52, State: Initialized, Role: FOLLOWER
I20260926 06:29:28.059851 24934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } }
I20260926 06:29:28.060156 25055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } } }
I20260926 06:29:28.060801 25055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:28.060192 24934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 06:29:28.060338 24908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b3fc0500e3643678744f4b25408b6a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e107df297c34572813ef9c895018b52"
I20260926 06:29:28.060946 24908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [term 0 FOLLOWER]: Advancing to term 1
I20260926 06:29:28.061851 24908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b3fc0500e3643678744f4b25408b6a4 in term 1.
I20260926 06:29:28.062672 24983 raft_consensus.cc:1261] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [term 1 FOLLOWER]: Refusing update from remote peer 1b3fc0500e3643678744f4b25408b6a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:29:28.062964 24908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [term 1 FOLLOWER]: Refusing update from remote peer 1b3fc0500e3643678744f4b25408b6a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 06:29:28.063128 25050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:29:28.063242 25016 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 06:29:28.065515 25030 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b3fc0500e3643678744f4b25408b6a4. Latest consensus state: current_term: 1 leader_uuid: "1b3fc0500e3643678744f4b25408b6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } } }
I20260926 06:29:28.065629 25030 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:28.065945 25055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b3fc0500e3643678744f4b25408b6a4. Latest consensus state: current_term: 1 leader_uuid: "1b3fc0500e3643678744f4b25408b6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } } }
I20260926 06:29:28.066113 25055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:28.068800 25016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b3fc0500e3643678744f4b25408b6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } } }
I20260926 06:29:28.068898 25016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:28.068989 25016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b3fc0500e3643678744f4b25408b6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } } }
I20260926 06:29:28.069039 25016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4 [sys.catalog]: This master's current role is: LEADER
I20260926 06:29:28.069379 25055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b3fc0500e3643678744f4b25408b6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } } }
I20260926 06:29:28.069461 25055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 06:29:28.069766 25056 mvcc.cc:205] Tried to move back new op lower bound from 7333495472381517824 to 7333495472254476288. Current Snapshot: MvccSnapshot[applied={T|T < 7333495472381517824}]
I20260926 06:29:28.070350 25030 sys_catalog.cc:455] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b3fc0500e3643678744f4b25408b6a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b3fc0500e3643678744f4b25408b6a4" member_type: VOTER last_known_addr { host: "127.2.39.254" port: 42369 } } peers { permanent_uuid: "7e107df297c34572813ef9c895018b52" member_type: VOTER last_known_addr { host: "127.2.39.253" port: 42003 } } peers { permanent_uuid: "acc593f2d16b40ea820f2dd4a4a197ab" member_type: VOTER last_known_addr { host: "127.2.39.252" port: 38769 } } }
I20260926 06:29:28.070430 25030 sys_catalog.cc:458] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab [sys.catalog]: This master's current role is: FOLLOWER
W20260926 06:29:28.070971 25070 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 06:29:28.071022 25070 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 06:29:28.074025 25052 catalog_manager.cc:1347] Generated new cluster ID: f78df81f6e22484b8b86a1178583780d
I20260926 06:29:28.074100 25052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 06:29:28.092132 25052 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 06:29:28.092916 25052 catalog_manager.cc:1504] Loading token signing keys...
I20260926 06:29:28.116389 25052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b3fc0500e3643678744f4b25408b6a4: Generated new TSK 0
I20260926 06:29:28.116612 25052 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 06:29:28.143913 24862 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.142637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.2.39.254:42369)
0926 06:29:28.142683 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.142685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.143185 (+ 500us) server_negotiation.cc:407] Connection header received
0926 06:29:28.143298 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.143299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.143311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.143345 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.143834 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43084: BlockingRecv error: recv got EOF from 127.0.0.1:43084 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 06:29:28.144982 25014 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.143044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.2.39.252:38769)
0926 06:29:28.143100 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.143102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.144400 (+ 1298us) server_negotiation.cc:407] Connection header received
0926 06:29:28.144545 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.144547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.144556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.144586 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.144920 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59434: BlockingRecv error: recv got EOF from 127.0.0.1:59434 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 06:29:28.145918 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.144214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.2.39.253:42003)
0926 06:29:28.144255 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.144257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.145188 (+ 931us) server_negotiation.cc:407] Connection header received
0926 06:29:28.145287 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.145288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.145299 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.145330 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.145837 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35288: BlockingRecv error: recv got EOF from 127.0.0.1:35288 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 06:29:28.148218 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.147468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.2.39.253:42003)
0926 06:29:28.147502 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.147503 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.147685 (+ 182us) server_negotiation.cc:407] Connection header received
0926 06:29:28.147790 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.147791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.147821 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.147851 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.148165 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35304: BlockingRecv error: recv got EOF from 127.0.0.1:35304 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 06:29:28.151283 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.150534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.2.39.253:42003)
0926 06:29:28.150566 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.150568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.150776 (+ 208us) server_negotiation.cc:407] Connection header received
0926 06:29:28.150888 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.150889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.150899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.150929 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.151237 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35312: BlockingRecv error: recv got EOF from 127.0.0.1:35312 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 06:29:28.155313 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.154491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.2.39.253:42003)
0926 06:29:28.154523 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.154525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.154760 (+ 235us) server_negotiation.cc:407] Connection header received
0926 06:29:28.154872 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.154873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.154884 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.154914 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.155268 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35320: BlockingRecv error: recv got EOF from 127.0.0.1:35320 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 06:29:28.161602 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.160630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.2.39.253:42003)
0926 06:29:28.160670 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.160672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.160928 (+ 256us) server_negotiation.cc:407] Connection header received
0926 06:29:28.161034 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.161036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.161047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.161096 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.161515 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35326: BlockingRecv error: recv got EOF from 127.0.0.1:35326 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 06:29:28.172041 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.171083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.2.39.253:42003)
0926 06:29:28.171131 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.171133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.171422 (+ 289us) server_negotiation.cc:407] Connection header received
0926 06:29:28.171530 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.171532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.171543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.171574 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.171975 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35330: BlockingRecv error: recv got EOF from 127.0.0.1:35330 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 06:29:28.190474 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.189471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.2.39.253:42003)
0926 06:29:28.189515 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.189518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.189836 (+ 318us) server_negotiation.cc:407] Connection header received
0926 06:29:28.189951 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.189952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.189964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.189996 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.190412 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35332: BlockingRecv error: recv got EOF from 127.0.0.1:35332 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 06:29:28.224858 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.223958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.2.39.253:42003)
0926 06:29:28.224029 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.224031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.224197 (+ 166us) server_negotiation.cc:407] Connection header received
0926 06:29:28.224364 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.224365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.224377 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.224452 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.224799 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35348: BlockingRecv error: recv got EOF from 127.0.0.1:35348 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 06:29:28.290838 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.290118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.2.39.253:42003)
0926 06:29:28.290189 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.290192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.290295 (+ 103us) server_negotiation.cc:407] Connection header received
0926 06:29:28.290345 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.290346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.290358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.290413 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.290776 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35352: BlockingRecv error: recv got EOF from 127.0.0.1:35352 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 06:29:28.420869 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.420051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53498 (local address 127.2.39.253:42003)
0926 06:29:28.420125 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.420127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.420225 (+ 98us) server_negotiation.cc:407] Connection header received
0926 06:29:28.420322 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.420323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.420338 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.420407 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.420782 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53498: BlockingRecv error: recv got EOF from 127.0.0.1:53498 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 06:29:28.678960 24938 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 06:29:28.678203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.2.39.253:42003)
0926 06:29:28.678275 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:28.678278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:28.678357 (+ 79us) server_negotiation.cc:407] Connection header received
0926 06:29:28.678442 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:28.678443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:28.678456 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:28.678522 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 06:29:28.678885 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53514: BlockingRecv error: recv got EOF from 127.0.0.1:53514 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 06:29:29.026010 25041 catalog_manager.cc:1259] Loaded cluster ID: f78df81f6e22484b8b86a1178583780d
I20260926 06:29:29.026088 25041 catalog_manager.cc:1552] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab: loading cluster ID for follower catalog manager: success
I20260926 06:29:29.027012 25041 catalog_manager.cc:1574] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab: acquiring CA information for follower catalog manager: success
I20260926 06:29:29.027374 25041 catalog_manager.cc:1602] T 00000000000000000000000000000000 P acc593f2d16b40ea820f2dd4a4a197ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:29:29.071524 25070 catalog_manager.cc:1259] Loaded cluster ID: f78df81f6e22484b8b86a1178583780d
I20260926 06:29:29.071646 25070 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52: loading cluster ID for follower catalog manager: success
I20260926 06:29:29.072467 25070 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52: acquiring CA information for follower catalog manager: success
I20260926 06:29:29.072793 25070 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e107df297c34572813ef9c895018b52: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 06:29:29.195062 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:29.192136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.2.39.253:42003)
0926 06:29:29.192252 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:29.192255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:29.192336 (+ 81us) server_negotiation.cc:407] Connection header received
0926 06:29:29.192462 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:29.192463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:29.192514 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:29.192611 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:29:29.192778 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:29.193275 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:29.194171 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:29.194373 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:29.194468 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:29.194675 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:29:29.194786 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:29:29.194874 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:29.194930 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 06:29:29.194951 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260926 06:29:29.199558 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:29.196926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.2.39.252:38769)
0926 06:29:29.197040 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:29.197042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:29.197096 (+ 54us) server_negotiation.cc:407] Connection header received
0926 06:29:29.197169 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:29.197169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:29.197206 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:29.197277 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:29:29.197397 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:29.198023 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:29.198783 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:29.198993 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:29.199049 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:29.199160 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:29:29.199278 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:29:29.199335 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:29.199429 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 06:29:29.199446 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260926 06:29:29.204625 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:29.201364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53532 (local address 127.2.39.253:42003)
0926 06:29:29.201409 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:29.201411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:29.201627 (+ 216us) server_negotiation.cc:407] Connection header received
0926 06:29:29.201761 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:29.201762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:29.201777 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:29.201836 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:29:29.201953 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:29.202710 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:29.203861 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:29.204076 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:29.204126 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:29.204289 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:29:29.204335 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:29:29.204392 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:29.204503 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 06:29:29.204535 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 06:29:29.204720 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:29.201710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.2.39.252:38769)
0926 06:29:29.201750 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:29.201752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:29.201761 (+ 9us) server_negotiation.cc:407] Connection header received
0926 06:29:29.201798 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:29.201799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:29.201809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:29.201855 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:29:29.202012 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:29.202712 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:29.203893 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:29.204125 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:29.204191 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:29.204354 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:29:29.204383 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:29:29.204444 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:29.204534 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 06:29:29.204562 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 06:29:29.204741 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 06:29:29.201369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.2.39.254:42369)
0926 06:29:29.201850 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 06:29:29.201853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 06:29:29.201860 (+ 7us) server_negotiation.cc:407] Connection header received
0926 06:29:29.201905 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 06:29:29.201906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 06:29:29.201959 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 06:29:29.202038 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 06:29:29.202199 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:29.202692 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:29.203481 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 06:29:29.203683 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 06:29:29.203765 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 06:29:29.204278 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 06:29:29.204395 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 06:29:29.204453 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 06:29:29.204616 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 06:29:29.204642 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":56,"threads_started":1}
I20260926 06:29:29.206439 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 24791
I20260926 06:29:29.212380 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 24866
I20260926 06:29:29.217700 2207 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOgNaKF/build/release/bin/kudu with pid 24941
2026-09-26T06:29:29Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2875 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5706 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269864 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST